2026-02-18T06:14:35.001 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-18T06:14:35.017 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-18T06:14:35.095 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885 branch: 20.2.1_2.17.26-baseline_2 description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-policy-none} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '54885' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror.sh workunit to test the rbd-mirror daemon in journal mode name: yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: 20.2.1_2.17.26-baseline_2 ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: 20.2.1_2.17.26-baseline_2 sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 7878 sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: 20.2.1_2.17.26-baseline_2 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed targets: trial031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHRNW7A2DvvWlKHRZNI6CLZ97SvCZ/eAtdF852Rm9tFeUL7CxpN5UeNDcxfpYyTzxAc0h0SLbexV5u04UgbTlx4= trial130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEwrfL57bUfT56dxS9dPm4meFgSKQcTf5bVd/MCniJ+6npZE+yMJCdieVstdiz0p0rICGb+5W5V0e2I6sZx+XsM= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror.sh env: CEPH_ARGS: '' RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: journal RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-17_21:26:40 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682 2026-02-18T06:14:35.095 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa; will attempt to use it 2026-02-18T06:14:35.096 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa/tasks 2026-02-18T06:14:35.096 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-18T06:14:35.097 INFO:teuthology.task.internal:Checking packages... 2026-02-18T06:14:35.097 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'adc1923395f2bcd6ae89b5d331d278364d1848ed' 2026-02-18T06:14:35.097 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-18T06:14:35.097 INFO:teuthology.packaging:ref: None 2026-02-18T06:14:35.097 INFO:teuthology.packaging:tag: None 2026-02-18T06:14:35.098 INFO:teuthology.packaging:branch: 20.2.1_2.17.26-baseline_2 2026-02-18T06:14:35.098 INFO:teuthology.packaging:sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T06:14:35.098 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=20.2.1_2.17.26-baseline_2 2026-02-18T06:14:35.217 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-670.gadc19233 2026-02-18T06:14:35.218 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-18T06:14:35.230 INFO:teuthology.task.internal:no buildpackages task found 2026-02-18T06:14:35.230 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-18T06:14:35.248 INFO:teuthology.task.internal:Saving configuration 2026-02-18T06:14:35.262 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-18T06:14:35.273 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-18T06:14:35.341 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial031.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-18 06:11:09.470316', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHRNW7A2DvvWlKHRZNI6CLZ97SvCZ/eAtdF852Rm9tFeUL7CxpN5UeNDcxfpYyTzxAc0h0SLbexV5u04UgbTlx4='} 2026-02-18T06:14:35.397 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial130.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-18 06:11:09.470931', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEwrfL57bUfT56dxS9dPm4meFgSKQcTf5bVd/MCniJ+6npZE+yMJCdieVstdiz0p0rICGb+5W5V0e2I6sZx+XsM='} 2026-02-18T06:14:35.397 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-18T06:14:35.407 INFO:teuthology.task.internal:roles: ubuntu@trial031.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2026-02-18T06:14:35.407 INFO:teuthology.task.internal:roles: ubuntu@trial130.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2026-02-18T06:14:35.407 INFO:teuthology.run_tasks:Running task console_log... 2026-02-18T06:14:35.539 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x70c907b9e520>, signals=[15]) 2026-02-18T06:14:35.539 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-18T06:14:35.548 INFO:teuthology.task.internal:Opening connections... 2026-02-18T06:14:35.548 DEBUG:teuthology.task.internal:connecting to ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:14:35.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:14:35.611 DEBUG:teuthology.task.internal:connecting to ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:14:35.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:14:35.670 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-18T06:14:35.681 DEBUG:teuthology.orchestra.run.trial031:> uname -m 2026-02-18T06:14:35.697 INFO:teuthology.orchestra.run.trial031.stdout:x86_64 2026-02-18T06:14:35.697 DEBUG:teuthology.orchestra.run.trial031:> cat /etc/os-release 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:NAME="CentOS Stream" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:VERSION="9" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:ID="centos" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:ID_LIKE="rhel fedora" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:VERSION_ID="9" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:PLATFORM_ID="platform:el9" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:ANSI_COLOR="0;31" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:LOGO="fedora-logo-icon" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:HOME_URL="https://centos.org/" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-18T06:14:35.753 INFO:teuthology.orchestra.run.trial031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-18T06:14:35.754 INFO:teuthology.lock.ops:Updating trial031.front.sepia.ceph.com on lock server 2026-02-18T06:14:35.826 DEBUG:teuthology.orchestra.run.trial130:> uname -m 2026-02-18T06:14:35.841 INFO:teuthology.orchestra.run.trial130.stdout:x86_64 2026-02-18T06:14:35.841 DEBUG:teuthology.orchestra.run.trial130:> cat /etc/os-release 2026-02-18T06:14:35.896 INFO:teuthology.orchestra.run.trial130.stdout:NAME="CentOS Stream" 2026-02-18T06:14:35.896 INFO:teuthology.orchestra.run.trial130.stdout:VERSION="9" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:ID="centos" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:ID_LIKE="rhel fedora" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:VERSION_ID="9" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:PLATFORM_ID="platform:el9" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:ANSI_COLOR="0;31" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:LOGO="fedora-logo-icon" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:HOME_URL="https://centos.org/" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-18T06:14:35.897 INFO:teuthology.orchestra.run.trial130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-18T06:14:35.897 INFO:teuthology.lock.ops:Updating trial130.front.sepia.ceph.com on lock server 2026-02-18T06:14:35.974 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-18T06:14:35.986 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-18T06:14:35.998 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-18T06:14:35.998 DEBUG:teuthology.orchestra.run.trial031:> test '!' -e /home/ubuntu/cephtest 2026-02-18T06:14:36.000 DEBUG:teuthology.orchestra.run.trial130:> test '!' -e /home/ubuntu/cephtest 2026-02-18T06:14:36.015 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-18T06:14:36.026 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-18T06:14:36.026 DEBUG:teuthology.orchestra.run.trial031:> test -z $(ls -A /var/lib/ceph) 2026-02-18T06:14:36.055 DEBUG:teuthology.orchestra.run.trial130:> test -z $(ls -A /var/lib/ceph) 2026-02-18T06:14:36.072 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-18T06:14:36.233 INFO:teuthology.run_tasks:Running task kernel... 2026-02-18T06:14:36.251 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-18T06:14:36.251 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-18T06:14:36.251 DEBUG:teuthology.orchestra.run.trial031:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-18T06:14:36.251 DEBUG:teuthology.orchestra.run.trial130:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-18T06:14:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:14:36.266 DEBUG:teuthology.orchestra.run.trial031:> uname -r 2026-02-18T06:14:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:14:36.266 DEBUG:teuthology.orchestra.run.trial130:> uname -r 2026-02-18T06:14:36.321 INFO:teuthology.orchestra.run.trial031.stdout:5.14.0-645.el9.x86_64 2026-02-18T06:14:36.322 INFO:teuthology.task.kernel:Running kernel on trial031: 5.14.0-645.el9.x86_64 2026-02-18T06:14:36.322 DEBUG:teuthology.orchestra.run.trial031:> sudo yum install -y kernel 2026-02-18T06:14:36.322 INFO:teuthology.orchestra.run.trial130.stdout:5.14.0-645.el9.x86_64 2026-02-18T06:14:36.322 INFO:teuthology.task.kernel:Running kernel on trial130: 5.14.0-645.el9.x86_64 2026-02-18T06:14:36.322 DEBUG:teuthology.orchestra.run.trial130:> sudo yum install -y kernel 2026-02-18T06:14:37.951 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.9 MB 00:01 2026-02-18T06:14:40.925 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-18T06:14:44.738 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 7.8 MB 00:01 2026-02-18T06:14:44.785 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.9 MB 00:08 2026-02-18T06:14:46.560 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-18T06:14:47.197 INFO:teuthology.orchestra.run.trial130.stdout:Extra Packages for Enterprise Linux 35 MB/s | 20 MB 00:00 2026-02-18T06:14:47.998 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-18T06:14:50.361 INFO:teuthology.orchestra.run.trial130.stdout:lab-extras 1.6 MB/s | 50 kB 00:00 2026-02-18T06:14:51.370 INFO:teuthology.orchestra.run.trial130.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-18T06:14:51.392 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:14:51.393 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:14:51.393 INFO:teuthology.orchestra.run.trial130.stdout: Package Architecture Version Repository Size 2026-02-18T06:14:51.393 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:14:51.393 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:14:51.393 INFO:teuthology.orchestra.run.trial130.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout:Installing dependencies: 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout: kernel-core x86_64 5.14.0-681.el9 baseos 17 M 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout: kernel-modules x86_64 5.14.0-681.el9 baseos 40 M 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout: kernel-modules-core x86_64 5.14.0-681.el9 baseos 31 M 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout:Install 4 Packages 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 88 M 2026-02-18T06:14:51.394 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 135 M 2026-02-18T06:14:51.395 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:14:51.513 INFO:teuthology.orchestra.run.trial130.stdout:(1/4): kernel-5.14.0-681.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2026-02-18T06:14:51.657 INFO:teuthology.orchestra.run.trial130.stdout:(2/4): kernel-core-5.14.0-681.el9.x86_64.rpm 85 MB/s | 17 MB 00:00 2026-02-18T06:14:51.944 INFO:teuthology.orchestra.run.trial130.stdout:(3/4): kernel-modules-5.14.0-681.el9.x86_64.rpm 81 MB/s | 40 MB 00:00 2026-02-18T06:14:52.252 INFO:teuthology.orchestra.run.trial130.stdout:(4/4): kernel-modules-core-5.14.0-681.el9.x86_6 42 MB/s | 31 MB 00:00 2026-02-18T06:14:52.252 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:14:52.252 INFO:teuthology.orchestra.run.trial130.stdout:Total 103 MB/s | 88 MB 00:00 2026-02-18T06:14:52.444 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:14:52.646 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:14:52.647 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:14:52.993 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:14:52.994 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:14:54.078 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:14:54.358 INFO:teuthology.orchestra.run.trial130.stdout: Installing : kernel-modules-core-5.14.0-681.el9.x86_64 1/4 2026-02-18T06:14:54.378 INFO:teuthology.orchestra.run.trial130.stdout: Installing : kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-18T06:14:56.164 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-18T06:14:56.193 INFO:teuthology.orchestra.run.trial130.stdout: Installing : kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-18T06:14:58.342 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - CRB 997 kB/s | 7.8 MB 00:07 2026-02-18T06:14:59.217 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - Extras packages 132 kB/s | 20 kB 00:00 2026-02-18T06:14:59.435 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-18T06:14:59.441 INFO:teuthology.orchestra.run.trial130.stdout: Installing : kernel-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:14:59.479 INFO:teuthology.orchestra.run.trial031.stdout:Extra Packages for Enterprise Linux 101 MB/s | 20 MB 00:00 2026-02-18T06:15:02.479 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: kernel-modules-core-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:02.678 INFO:teuthology.orchestra.run.trial031.stdout:lab-extras 808 kB/s | 50 kB 00:00 2026-02-18T06:15:03.685 INFO:teuthology.orchestra.run.trial031.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-18T06:15:03.707 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout: Package Architecture Version Repository Size 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout:Installing dependencies: 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout: kernel-core x86_64 5.14.0-681.el9 baseos 17 M 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout: kernel-modules x86_64 5.14.0-681.el9 baseos 40 M 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout: kernel-modules-core x86_64 5.14.0-681.el9 baseos 31 M 2026-02-18T06:15:03.709 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:15:03.710 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:15:03.710 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:15:03.710 INFO:teuthology.orchestra.run.trial031.stdout:Install 4 Packages 2026-02-18T06:15:03.710 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:15:03.710 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 88 M 2026-02-18T06:15:03.710 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 135 M 2026-02-18T06:15:03.710 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:15:04.418 INFO:teuthology.orchestra.run.trial031.stdout:(1/4): kernel-5.14.0-681.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2026-02-18T06:15:04.988 INFO:teuthology.orchestra.run.trial031.stdout:(2/4): kernel-core-5.14.0-681.el9.x86_64.rpm 14 MB/s | 17 MB 00:01 2026-02-18T06:15:05.500 INFO:teuthology.orchestra.run.trial031.stdout:(3/4): kernel-modules-core-5.14.0-681.el9.x86_6 29 MB/s | 31 MB 00:01 2026-02-18T06:15:05.645 INFO:teuthology.orchestra.run.trial031.stdout:(4/4): kernel-modules-5.14.0-681.el9.x86_64.rpm 21 MB/s | 40 MB 00:01 2026-02-18T06:15:05.645 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:15:05.645 INFO:teuthology.orchestra.run.trial031.stdout:Total 46 MB/s | 88 MB 00:01 2026-02-18T06:15:05.841 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:15:06.045 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:15:06.045 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:15:06.394 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:15:06.394 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:15:07.478 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:15:07.759 INFO:teuthology.orchestra.run.trial031.stdout: Installing : kernel-modules-core-5.14.0-681.el9.x86_64 1/4 2026-02-18T06:15:07.778 INFO:teuthology.orchestra.run.trial031.stdout: Installing : kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-18T06:15:09.562 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-18T06:15:09.590 INFO:teuthology.orchestra.run.trial031.stdout: Installing : kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-18T06:15:12.672 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-18T06:15:12.678 INFO:teuthology.orchestra.run.trial031.stdout: Installing : kernel-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:15.711 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: kernel-modules-core-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:32.875 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: kernel-core-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:32.896 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: kernel-modules-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:32.961 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:32.961 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/4 2026-02-18T06:15:32.961 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-18T06:15:32.961 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-18T06:15:33.012 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : kernel-modules-core-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout: kernel-core-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout: kernel-modules-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout: kernel-modules-core-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:15:33.013 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:15:33.071 DEBUG:teuthology.orchestra.run.trial130:> rpm -q kernel | sort -rV | head -n 1 2026-02-18T06:15:33.096 INFO:teuthology.orchestra.run.trial130.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.097 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-18T06:15:33.098 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-681.el9.x86_64 2026-02-18T06:15:33.098 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-18T06:15:33.098 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-18T06:15:33.098 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2026-02-18T06:15:33.098 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial130.front.sepia.ceph.com, path=None, version=distro) 2026-02-18T06:15:33.098 DEBUG:teuthology.orchestra.run.trial130:> rpm -q kernel | sort -rV | head -n 1 2026-02-18T06:15:33.161 INFO:teuthology.orchestra.run.trial130.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.162 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-18T06:15:33.162 DEBUG:teuthology.orchestra.run.trial130:> sudo rpm -qi grub2-tools 2026-02-18T06:15:33.231 INFO:teuthology.orchestra.run.trial130.stdout:Name : grub2-tools 2026-02-18T06:15:33.231 INFO:teuthology.orchestra.run.trial130.stdout:Epoch : 1 2026-02-18T06:15:33.231 INFO:teuthology.orchestra.run.trial130.stdout:Version : 2.06 2026-02-18T06:15:33.231 INFO:teuthology.orchestra.run.trial130.stdout:Release : 118.el9 2026-02-18T06:15:33.231 INFO:teuthology.orchestra.run.trial130.stdout:Architecture: x86_64 2026-02-18T06:15:33.231 INFO:teuthology.orchestra.run.trial130.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-18T06:15:33.231 INFO:teuthology.orchestra.run.trial130.stdout:Group : Unspecified 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Size : 8393909 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:License : GPLv3+ 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Packager : builder@centos.org 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Vendor : CentOS 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Summary : Support tools for GRUB. 2026-02-18T06:15:33.232 INFO:teuthology.orchestra.run.trial130.stdout:Description : 2026-02-18T06:15:33.233 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:15:33.233 INFO:teuthology.orchestra.run.trial130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-18T06:15:33.233 INFO:teuthology.orchestra.run.trial130.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-18T06:15:33.233 INFO:teuthology.orchestra.run.trial130.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-18T06:15:33.233 INFO:teuthology.orchestra.run.trial130.stdout:hardware devices. 2026-02-18T06:15:33.233 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:15:33.233 INFO:teuthology.orchestra.run.trial130.stdout:This subpackage provides tools for support of all platforms. 2026-02-18T06:15:33.234 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-18T06:15:33.234 INFO:teuthology.task.kernel:Updating grub on trial130 to boot 5.14.0-681.el9.x86_64 2026-02-18T06:15:33.235 DEBUG:teuthology.orchestra.run.trial130:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-18T06:15:33.300 INFO:teuthology.orchestra.run.trial130.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-18T06:15:33.302 DEBUG:teuthology.orchestra.run.trial130:> sudo command -v grubby && echo yes || echo no 2026-02-18T06:15:33.366 INFO:teuthology.orchestra.run.trial130.stdout:/sbin/grubby 2026-02-18T06:15:33.368 INFO:teuthology.orchestra.run.trial130.stdout:yes 2026-02-18T06:15:33.368 DEBUG:teuthology.orchestra.run.trial130:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-681.el9.x86_64" | head -n 1 2026-02-18T06:15:33.433 INFO:teuthology.orchestra.run.trial130.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-18T06:15:33.434 DEBUG:teuthology.orchestra.run.trial130:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.507 DEBUG:teuthology.orchestra.run.trial130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-18T06:15:33.714 INFO:teuthology.orchestra.run.trial130.stderr:Generating grub configuration file ... 2026-02-18T06:15:33.779 INFO:teuthology.orchestra.run.trial130.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-18T06:15:33.785 INFO:teuthology.orchestra.run.trial130.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-18T06:15:34.034 INFO:teuthology.orchestra.run.trial130.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-18T06:15:34.041 INFO:teuthology.orchestra.run.trial130.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-18T06:15:34.073 INFO:teuthology.orchestra.run.trial130.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-18T06:15:34.079 INFO:teuthology.orchestra.run.trial130.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-18T06:15:34.102 INFO:teuthology.orchestra.run.trial130.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-18T06:15:34.109 INFO:teuthology.orchestra.run.trial130.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-18T06:15:34.190 INFO:teuthology.orchestra.run.trial130.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-18T06:15:34.198 INFO:teuthology.orchestra.run.trial130.stderr:done 2026-02-18T06:15:34.201 DEBUG:teuthology.orchestra.run.trial130:> test -d /sys/firmware/efi 2026-02-18T06:15:34.216 DEBUG:teuthology.orchestra.run.trial130:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-18T06:15:34.282 INFO:teuthology.orchestra.run.trial130.stdout:/boot/efi/EFI/BOOT 2026-02-18T06:15:34.282 INFO:teuthology.orchestra.run.trial130.stdout:/boot/efi/EFI/centos 2026-02-18T06:15:34.284 DEBUG:teuthology.orchestra.run.trial130:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-18T06:15:34.350 INFO:teuthology.orchestra.run.trial130.stdout:no 2026-02-18T06:15:34.351 DEBUG:teuthology.orchestra.run.trial130:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-18T06:15:34.416 INFO:teuthology.orchestra.run.trial130.stdout:yes 2026-02-18T06:15:34.416 DEBUG:teuthology.orchestra.run.trial130:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-18T06:15:34.481 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial130 2026-02-18T06:15:34.481 DEBUG:teuthology.orchestra.run.trial130:> sudo bash -c ' 2026-02-18T06:15:34.481 DEBUG:teuthology.orchestra.run.trial130:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-18T06:15:34.481 DEBUG:teuthology.orchestra.run.trial130:> ' 2026-02-18T06:15:45.959 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: kernel-core-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:45.979 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: kernel-modules-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:46.151 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:46.152 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/4 2026-02-18T06:15:46.152 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : kernel-core-5.14.0-681.el9.x86_64 2/4 2026-02-18T06:15:46.152 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : kernel-modules-5.14.0-681.el9.x86_64 3/4 2026-02-18T06:15:46.203 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : kernel-modules-core-5.14.0-681.el9.x86_64 4/4 2026-02-18T06:15:46.203 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:15:46.203 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:15:46.204 INFO:teuthology.orchestra.run.trial031.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.204 INFO:teuthology.orchestra.run.trial031.stdout: kernel-core-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.204 INFO:teuthology.orchestra.run.trial031.stdout: kernel-modules-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.204 INFO:teuthology.orchestra.run.trial031.stdout: kernel-modules-core-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.204 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:15:46.204 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:15:46.259 DEBUG:teuthology.orchestra.run.trial031:> rpm -q kernel | sort -rV | head -n 1 2026-02-18T06:15:46.284 INFO:teuthology.orchestra.run.trial031.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.285 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-18T06:15:46.286 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-681.el9.x86_64 2026-02-18T06:15:46.286 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-18T06:15:46.286 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-18T06:15:46.286 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2026-02-18T06:15:46.286 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial031.front.sepia.ceph.com, path=None, version=distro) 2026-02-18T06:15:46.286 DEBUG:teuthology.orchestra.run.trial031:> rpm -q kernel | sort -rV | head -n 1 2026-02-18T06:15:46.348 INFO:teuthology.orchestra.run.trial031.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.349 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-18T06:15:46.350 DEBUG:teuthology.orchestra.run.trial031:> sudo rpm -qi grub2-tools 2026-02-18T06:15:46.417 INFO:teuthology.orchestra.run.trial031.stdout:Name : grub2-tools 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Epoch : 1 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Version : 2.06 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Release : 118.el9 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Architecture: x86_64 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Group : Unspecified 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Size : 8393909 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:License : GPLv3+ 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-18T06:15:46.418 INFO:teuthology.orchestra.run.trial031.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:Packager : builder@centos.org 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:Vendor : CentOS 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:Summary : Support tools for GRUB. 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:Description : 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-18T06:15:46.419 INFO:teuthology.orchestra.run.trial031.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-18T06:15:46.420 INFO:teuthology.orchestra.run.trial031.stdout:hardware devices. 2026-02-18T06:15:46.420 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:15:46.420 INFO:teuthology.orchestra.run.trial031.stdout:This subpackage provides tools for support of all platforms. 2026-02-18T06:15:46.421 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-18T06:15:46.421 INFO:teuthology.task.kernel:Updating grub on trial031 to boot 5.14.0-681.el9.x86_64 2026-02-18T06:15:46.421 DEBUG:teuthology.orchestra.run.trial031:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-18T06:15:46.483 INFO:teuthology.orchestra.run.trial031.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-18T06:15:46.484 DEBUG:teuthology.orchestra.run.trial031:> sudo command -v grubby && echo yes || echo no 2026-02-18T06:15:46.548 INFO:teuthology.orchestra.run.trial031.stdout:/sbin/grubby 2026-02-18T06:15:46.550 INFO:teuthology.orchestra.run.trial031.stdout:yes 2026-02-18T06:15:46.551 DEBUG:teuthology.orchestra.run.trial031:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-681.el9.x86_64" | head -n 1 2026-02-18T06:15:46.614 INFO:teuthology.orchestra.run.trial031.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-18T06:15:46.615 DEBUG:teuthology.orchestra.run.trial031:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.689 DEBUG:teuthology.orchestra.run.trial031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-18T06:15:46.900 INFO:teuthology.orchestra.run.trial031.stderr:Generating grub configuration file ... 2026-02-18T06:15:46.961 INFO:teuthology.orchestra.run.trial031.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-18T06:15:46.967 INFO:teuthology.orchestra.run.trial031.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-18T06:15:47.221 INFO:teuthology.orchestra.run.trial031.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-18T06:15:47.227 INFO:teuthology.orchestra.run.trial031.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-18T06:15:47.259 INFO:teuthology.orchestra.run.trial031.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-18T06:15:47.265 INFO:teuthology.orchestra.run.trial031.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-18T06:15:47.289 INFO:teuthology.orchestra.run.trial031.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-18T06:15:47.295 INFO:teuthology.orchestra.run.trial031.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-18T06:15:47.377 INFO:teuthology.orchestra.run.trial031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-18T06:15:47.385 INFO:teuthology.orchestra.run.trial031.stderr:done 2026-02-18T06:15:47.387 DEBUG:teuthology.orchestra.run.trial031:> test -d /sys/firmware/efi 2026-02-18T06:15:47.403 DEBUG:teuthology.orchestra.run.trial031:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-18T06:15:47.467 INFO:teuthology.orchestra.run.trial031.stdout:/boot/efi/EFI/BOOT 2026-02-18T06:15:47.467 INFO:teuthology.orchestra.run.trial031.stdout:/boot/efi/EFI/centos 2026-02-18T06:15:47.469 DEBUG:teuthology.orchestra.run.trial031:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-18T06:15:47.533 INFO:teuthology.orchestra.run.trial031.stdout:no 2026-02-18T06:15:47.534 DEBUG:teuthology.orchestra.run.trial031:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-18T06:15:47.598 INFO:teuthology.orchestra.run.trial031.stdout:yes 2026-02-18T06:15:47.598 DEBUG:teuthology.orchestra.run.trial031:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-18T06:15:47.665 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial031 2026-02-18T06:15:47.665 DEBUG:teuthology.orchestra.run.trial031:> sudo bash -c ' 2026-02-18T06:15:47.665 DEBUG:teuthology.orchestra.run.trial031:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-18T06:15:47.665 DEBUG:teuthology.orchestra.run.trial031:> ' 2026-02-18T06:16:04.523 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2026-02-18T06:16:04.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial130.front.sepia.ceph.com' 2026-02-18T06:16:04.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:16:17.707 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2026-02-18T06:16:17.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial031.front.sepia.ceph.com' 2026-02-18T06:16:17.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:17:04.536 DEBUG:teuthology.orchestra.remote:timed out 2026-02-18T06:17:13.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial130.front.sepia.ceph.com' 2026-02-18T06:17:13.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:17:13.673 DEBUG:teuthology.orchestra.run.trial130:> true 2026-02-18T06:17:13.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial130.front.sepia.ceph.com' 2026-02-18T06:17:13.828 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-681.el9.x86_64"... 2026-02-18T06:17:13.828 DEBUG:teuthology.orchestra.run.trial130:> uname -r 2026-02-18T06:17:13.884 INFO:teuthology.orchestra.run.trial130.stdout:5.14.0-681.el9.x86_64 2026-02-18T06:17:13.884 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-681.el9.x86_64 vs 5.14.0-681.el9.x86_64 2026-02-18T06:17:13.885 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-18T06:17:13.885 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-18T06:17:14.885 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2026-02-18T06:17:14.886 DEBUG:teuthology.orchestra.run.trial130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-18T06:17:14.915 INFO:teuthology.orchestra.run.trial130.stdout:ttyS1 2026-02-18T06:17:14.944 DEBUG:teuthology.parallel:result is None 2026-02-18T06:17:17.711 DEBUG:teuthology.orchestra.remote:timed out 2026-02-18T06:17:26.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial031.front.sepia.ceph.com' 2026-02-18T06:17:26.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:17:26.847 DEBUG:teuthology.orchestra.run.trial031:> true 2026-02-18T06:17:27.001 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial031.front.sepia.ceph.com' 2026-02-18T06:17:27.001 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-681.el9.x86_64"... 2026-02-18T06:17:27.002 DEBUG:teuthology.orchestra.run.trial031:> uname -r 2026-02-18T06:17:27.057 INFO:teuthology.orchestra.run.trial031.stdout:5.14.0-681.el9.x86_64 2026-02-18T06:17:27.058 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-681.el9.x86_64 vs 5.14.0-681.el9.x86_64 2026-02-18T06:17:27.058 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-18T06:17:27.058 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-18T06:17:28.059 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2026-02-18T06:17:28.059 DEBUG:teuthology.orchestra.run.trial031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-18T06:17:28.089 INFO:teuthology.orchestra.run.trial031.stdout:ttyS1 2026-02-18T06:17:28.111 DEBUG:teuthology.parallel:result is None 2026-02-18T06:17:28.111 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-18T06:17:28.123 INFO:teuthology.task.internal:Creating test directory... 2026-02-18T06:17:28.123 DEBUG:teuthology.orchestra.run.trial031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-18T06:17:28.154 DEBUG:teuthology.orchestra.run.trial130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-18T06:17:28.172 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-18T06:17:28.183 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-18T06:17:28.203 INFO:teuthology.task.internal:Creating archive directory... 2026-02-18T06:17:28.203 DEBUG:teuthology.orchestra.run.trial031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-18T06:17:28.210 DEBUG:teuthology.orchestra.run.trial130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-18T06:17:28.240 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-18T06:17:28.252 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-18T06:17:28.252 DEBUG:teuthology.orchestra.run.trial031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-18T06:17:28.268 DEBUG:teuthology.orchestra.run.trial130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-18T06:17:28.290 INFO:teuthology.orchestra.run.trial031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T06:17:28.295 INFO:teuthology.orchestra.run.trial130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T06:17:28.298 INFO:teuthology.orchestra.run.trial031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T06:17:28.304 INFO:teuthology.orchestra.run.trial130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T06:17:28.306 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-18T06:17:28.318 INFO:teuthology.task.internal:Configuring sudo... 2026-02-18T06:17:28.318 DEBUG:teuthology.orchestra.run.trial031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-18T06:17:28.343 DEBUG:teuthology.orchestra.run.trial130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-18T06:17:28.369 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-18T06:17:28.386 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-18T06:17:28.386 DEBUG:teuthology.orchestra.run.trial031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-18T06:17:28.409 DEBUG:teuthology.orchestra.run.trial130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-18T06:17:28.426 DEBUG:teuthology.orchestra.run.trial031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T06:17:28.579 DEBUG:teuthology.orchestra.run.trial031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T06:17:28.605 DEBUG:teuthology.orchestra.run.trial031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T06:17:28.750 DEBUG:teuthology.orchestra.run.trial031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T06:17:28.775 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:17:28.775 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-18T06:17:28.840 DEBUG:teuthology.orchestra.run.trial130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T06:17:28.956 DEBUG:teuthology.orchestra.run.trial130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T06:17:28.981 DEBUG:teuthology.orchestra.run.trial130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T06:17:29.109 DEBUG:teuthology.orchestra.run.trial130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T06:17:29.134 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:17:29.134 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-18T06:17:29.198 DEBUG:teuthology.orchestra.run.trial031:> sudo service rsyslog restart 2026-02-18T06:17:29.200 DEBUG:teuthology.orchestra.run.trial130:> sudo service rsyslog restart 2026-02-18T06:17:29.226 INFO:teuthology.orchestra.run.trial031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-18T06:17:29.266 INFO:teuthology.orchestra.run.trial130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-18T06:17:29.710 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-18T06:17:29.722 INFO:teuthology.task.internal:Starting timer... 2026-02-18T06:17:29.723 INFO:teuthology.run_tasks:Running task pcp... 2026-02-18T06:17:29.745 INFO:teuthology.run_tasks:Running task selinux... 2026-02-18T06:17:29.762 DEBUG:teuthology.orchestra.run.trial031:> sudo service auditd rotate 2026-02-18T06:17:29.793 INFO:teuthology.orchestra.run.trial031.stdout:Rotating logs: 2026-02-18T06:17:29.795 DEBUG:teuthology.orchestra.run.trial130:> sudo service auditd rotate 2026-02-18T06:17:29.824 INFO:teuthology.orchestra.run.trial130.stdout:Rotating logs: 2026-02-18T06:17:29.826 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-18T06:17:29.826 DEBUG:teuthology.orchestra.run.trial031:> /usr/sbin/getenforce 2026-02-18T06:17:29.851 INFO:teuthology.orchestra.run.trial031.stdout:Permissive 2026-02-18T06:17:29.851 DEBUG:teuthology.orchestra.run.trial130:> /usr/sbin/getenforce 2026-02-18T06:17:29.883 INFO:teuthology.orchestra.run.trial130.stdout:Permissive 2026-02-18T06:17:29.884 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial031.front.sepia.ceph.com': 'permissive', 'ubuntu@trial130.front.sepia.ceph.com': 'permissive'} 2026-02-18T06:17:29.884 DEBUG:teuthology.orchestra.run.trial031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-18T06:17:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:17:29.915 DEBUG:teuthology.orchestra.run.trial130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-18T06:17:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:17:29.949 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-18T06:17:29.949 DEBUG:teuthology.orchestra.run.trial031:> sudo /usr/sbin/setenforce permissive 2026-02-18T06:17:29.980 DEBUG:teuthology.orchestra.run.trial130:> sudo /usr/sbin/setenforce permissive 2026-02-18T06:17:30.013 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-18T06:17:30.028 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-18T06:17:30.040 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-18T06:17:30.041 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial031.front.sepia.ceph.com,trial130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-18T06:18:45.224 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial031.front.sepia.ceph.com'), Remote(name='ubuntu@trial130.front.sepia.ceph.com')] 2026-02-18T06:18:45.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial031.front.sepia.ceph.com' 2026-02-18T06:18:45.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:18:45.335 DEBUG:teuthology.orchestra.run.trial031:> true 2026-02-18T06:18:45.409 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial031.front.sepia.ceph.com' 2026-02-18T06:18:45.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial130.front.sepia.ceph.com' 2026-02-18T06:18:45.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T06:18:45.500 DEBUG:teuthology.orchestra.run.trial130:> true 2026-02-18T06:18:45.573 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial130.front.sepia.ceph.com' 2026-02-18T06:18:45.573 INFO:teuthology.run_tasks:Running task clock... 2026-02-18T06:18:45.587 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-18T06:18:45.587 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-18T06:18:45.588 DEBUG:teuthology.orchestra.run.trial031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-18T06:18:45.590 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-18T06:18:45.590 DEBUG:teuthology.orchestra.run.trial130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-18T06:18:45.616 INFO:teuthology.orchestra.run.trial031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-18T06:18:45.628 INFO:teuthology.orchestra.run.trial031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-18T06:18:45.641 INFO:teuthology.orchestra.run.trial130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-18T06:18:45.647 INFO:teuthology.orchestra.run.trial031.stderr:sudo: ntpd: command not found 2026-02-18T06:18:45.653 INFO:teuthology.orchestra.run.trial130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-18T06:18:45.657 INFO:teuthology.orchestra.run.trial031.stdout:506 Cannot talk to daemon 2026-02-18T06:18:45.668 INFO:teuthology.orchestra.run.trial031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-18T06:18:45.674 INFO:teuthology.orchestra.run.trial130.stderr:sudo: ntpd: command not found 2026-02-18T06:18:45.680 INFO:teuthology.orchestra.run.trial031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-18T06:18:45.684 INFO:teuthology.orchestra.run.trial130.stdout:506 Cannot talk to daemon 2026-02-18T06:18:45.695 INFO:teuthology.orchestra.run.trial130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-18T06:18:45.707 INFO:teuthology.orchestra.run.trial130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-18T06:18:45.727 INFO:teuthology.orchestra.run.trial031.stderr:bash: line 1: ntpq: command not found 2026-02-18T06:18:45.756 INFO:teuthology.orchestra.run.trial130.stderr:bash: line 1: ntpq: command not found 2026-02-18T06:18:45.774 INFO:teuthology.orchestra.run.trial130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-18T06:18:45.774 INFO:teuthology.orchestra.run.trial130.stdout:=============================================================================== 2026-02-18T06:18:45.774 INFO:teuthology.orchestra.run.trial130.stdout:^? 72-14-182-49.ip.linodeus> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.774 INFO:teuthology.orchestra.run.trial130.stdout:^? 72-14-186-59.ip.linodeus> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.775 INFO:teuthology.orchestra.run.trial130.stdout:^? ntp2.wiktel.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.775 INFO:teuthology.orchestra.run.trial130.stdout:^? dns-e.ns4v.icu 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.827 INFO:teuthology.orchestra.run.trial031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-18T06:18:45.827 INFO:teuthology.orchestra.run.trial031.stdout:=============================================================================== 2026-02-18T06:18:45.827 INFO:teuthology.orchestra.run.trial031.stdout:^? li1150-42.members.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.827 INFO:teuthology.orchestra.run.trial031.stdout:^? ntp.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.827 INFO:teuthology.orchestra.run.trial031.stdout:^? 69.9.131.124 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.827 INFO:teuthology.orchestra.run.trial031.stdout:^? time.richie.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-18T06:18:45.828 INFO:teuthology.run_tasks:Running task install... 2026-02-18T06:18:45.843 DEBUG:teuthology.task.install:project ceph 2026-02-18T06:18:45.843 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}} 2026-02-18T06:18:45.843 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'} 2026-02-18T06:18:45.843 INFO:teuthology.task.install:Using flavor: default 2026-02-18T06:18:45.851 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-18T06:18:45.852 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2026-02-18T06:18:45.852 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed', 'tag': None, 'wait_for_package': False} 2026-02-18T06:18:45.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T06:18:45.856 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed', 'tag': None, 'wait_for_package': False} 2026-02-18T06:18:45.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T06:18:46.052 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/ 2026-02-18T06:18:46.052 INFO:teuthology.task.install.rpm:Package version is 20.2.0-670.gadc19233 2026-02-18T06:18:46.063 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/ 2026-02-18T06:18:46.063 INFO:teuthology.task.install.rpm:Package version is 20.2.0-670.gadc19233 2026-02-18T06:18:46.328 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-18T06:18:46.328 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:18:46.329 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-18T06:18:46.354 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2026-02-18T06:18:46.354 DEBUG:teuthology.orchestra.run.trial031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/adc1923395f2bcd6ae89b5d331d278364d1848ed/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-18T06:18:46.378 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-18T06:18:46.379 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:18:46.379 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-18T06:18:46.406 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2026-02-18T06:18:46.407 DEBUG:teuthology.orchestra.run.trial130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/adc1923395f2bcd6ae89b5d331d278364d1848ed/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-18T06:18:46.419 DEBUG:teuthology.orchestra.run.trial031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-18T06:18:46.471 DEBUG:teuthology.orchestra.run.trial130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-18T06:18:46.494 DEBUG:teuthology.orchestra.run.trial031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-18T06:18:46.544 DEBUG:teuthology.orchestra.run.trial130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-18T06:18:46.559 INFO:teuthology.orchestra.run.trial031.stdout:check_obsoletes = 1 2026-02-18T06:18:46.561 DEBUG:teuthology.orchestra.run.trial031:> sudo yum clean all 2026-02-18T06:18:46.607 INFO:teuthology.orchestra.run.trial130.stdout:check_obsoletes = 1 2026-02-18T06:18:46.609 DEBUG:teuthology.orchestra.run.trial130:> sudo yum clean all 2026-02-18T06:18:46.710 INFO:teuthology.orchestra.run.trial031.stdout:41 files removed 2026-02-18T06:18:46.725 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-radosgw 2026-02-18T06:18:46.757 INFO:teuthology.orchestra.run.trial130.stdout:41 files removed 2026-02-18T06:18:46.770 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-radosgw 2026-02-18T06:18:46.998 INFO:teuthology.orchestra.run.trial031.stdout:ceph packages for x86_64 693 kB/s | 89 kB 00:00 2026-02-18T06:18:47.042 INFO:teuthology.orchestra.run.trial130.stdout:ceph packages for x86_64 706 kB/s | 89 kB 00:00 2026-02-18T06:18:47.139 INFO:teuthology.orchestra.run.trial031.stdout:ceph noarch packages 162 kB/s | 19 kB 00:00 2026-02-18T06:18:47.184 INFO:teuthology.orchestra.run.trial130.stdout:ceph noarch packages 161 kB/s | 19 kB 00:00 2026-02-18T06:18:47.260 INFO:teuthology.orchestra.run.trial031.stdout:ceph source packages 19 kB/s | 1.9 kB 00:00 2026-02-18T06:18:47.311 INFO:teuthology.orchestra.run.trial130.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-18T06:18:47.599 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - BaseOS 28 MB/s | 8.9 MB 00:00 2026-02-18T06:18:48.402 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.9 MB 00:01 2026-02-18T06:18:48.651 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - AppStream 51 MB/s | 27 MB 00:00 2026-02-18T06:18:49.291 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - AppStream 76 MB/s | 27 MB 00:00 2026-02-18T06:18:52.824 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 7.8 MB 00:01 2026-02-18T06:18:53.564 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 7.8 MB 00:02 2026-02-18T06:18:54.433 INFO:teuthology.orchestra.run.trial130.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 20 kB 00:00 2026-02-18T06:18:54.784 INFO:teuthology.orchestra.run.trial130.stdout:Extra Packages for Enterprise Linux 70 MB/s | 20 MB 00:00 2026-02-18T06:18:55.381 INFO:teuthology.orchestra.run.trial031.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-18T06:18:56.240 INFO:teuthology.orchestra.run.trial031.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2026-02-18T06:18:58.037 INFO:teuthology.orchestra.run.trial130.stdout:lab-extras 920 kB/s | 50 kB 00:00 2026-02-18T06:18:59.262 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:18:59.263 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:18:59.263 INFO:teuthology.orchestra.run.trial130.stdout: Package Arch Version Repository Size 2026-02-18T06:18:59.263 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: ceph-radosgw x86_64 2:20.2.0-670.gadc19233.el9 ceph 24 M 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout:Upgrading: 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: librados2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 3.5 M 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: librbd1 x86_64 2:20.2.0-670.gadc19233.el9 ceph 2.8 M 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout:Installing dependencies: 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: ceph-base x86_64 2:20.2.0-670.gadc19233.el9 ceph 5.9 M 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: ceph-common x86_64 2:20.2.0-670.gadc19233.el9 ceph 24 M 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: ceph-selinux x86_64 2:20.2.0-670.gadc19233.el9 ceph 25 k 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-18T06:18:59.264 INFO:teuthology.orchestra.run.trial130.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 869 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: libradosstriper1 x86_64 2:20.2.0-670.gadc19233.el9 ceph 250 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: librgw2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 6.4 M 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-18T06:18:59.265 INFO:teuthology.orchestra.run.trial130.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: python3-ceph-argparse x86_64 2:20.2.0-670.gadc19233.el9 ceph 45 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: python3-ceph-common x86_64 2:20.2.0-670.gadc19233.el9 ceph 175 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: python3-cephfs x86_64 2:20.2.0-670.gadc19233.el9 ceph 162 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: python3-rados x86_64 2:20.2.0-670.gadc19233.el9 ceph 323 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: python3-rbd x86_64 2:20.2.0-670.gadc19233.el9 ceph 305 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: python3-rgw x86_64 2:20.2.0-670.gadc19233.el9 ceph 99 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-18T06:18:59.266 INFO:teuthology.orchestra.run.trial130.stdout:Installing weak dependencies: 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs-daemon x86_64 2:20.2.0-670.gadc19233.el9 ceph 35 k 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs-proxy2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 24 k 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout:Install 35 Packages 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout:Upgrade 2 Packages 2026-02-18T06:18:59.267 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:18:59.268 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 79 M 2026-02-18T06:18:59.268 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:18:59.440 INFO:teuthology.orchestra.run.trial031.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-18T06:18:59.762 INFO:teuthology.orchestra.run.trial130.stdout:(1/37): ceph-base-20.2.0-670.gadc19233.el9.x86_ 21 MB/s | 5.9 MB 00:00 2026-02-18T06:18:59.789 INFO:teuthology.orchestra.run.trial130.stdout:(2/37): ceph-selinux-20.2.0-670.gadc19233.el9.x 955 kB/s | 25 kB 00:00 2026-02-18T06:18:59.819 INFO:teuthology.orchestra.run.trial130.stdout:(3/37): libcephfs-daemon-20.2.0-670.gadc19233.e 1.2 MB/s | 35 kB 00:00 2026-02-18T06:18:59.850 INFO:teuthology.orchestra.run.trial130.stdout:(4/37): libcephfs-proxy2-20.2.0-670.gadc19233.e 777 kB/s | 24 kB 00:00 2026-02-18T06:18:59.901 INFO:teuthology.orchestra.run.trial130.stdout:(5/37): libcephfs2-20.2.0-670.gadc19233.el9.x86 17 MB/s | 869 kB 00:00 2026-02-18T06:18:59.940 INFO:teuthology.orchestra.run.trial130.stdout:(6/37): libradosstriper1-20.2.0-670.gadc19233.e 6.3 MB/s | 250 kB 00:00 2026-02-18T06:19:00.255 INFO:teuthology.orchestra.run.trial130.stdout:(7/37): librgw2-20.2.0-670.gadc19233.el9.x86_64 20 MB/s | 6.4 MB 00:00 2026-02-18T06:19:00.283 INFO:teuthology.orchestra.run.trial130.stdout:(8/37): python3-ceph-argparse-20.2.0-670.gadc19 1.6 MB/s | 45 kB 00:00 2026-02-18T06:19:00.317 INFO:teuthology.orchestra.run.trial130.stdout:(9/37): python3-ceph-common-20.2.0-670.gadc1923 5.0 MB/s | 175 kB 00:00 2026-02-18T06:19:00.351 INFO:teuthology.orchestra.run.trial130.stdout:(10/37): python3-cephfs-20.2.0-670.gadc19233.el 4.6 MB/s | 162 kB 00:00 2026-02-18T06:19:00.388 INFO:teuthology.orchestra.run.trial130.stdout:(11/37): python3-rados-20.2.0-670.gadc19233.el9 8.6 MB/s | 323 kB 00:00 2026-02-18T06:19:00.425 INFO:teuthology.orchestra.run.trial130.stdout:(12/37): python3-rbd-20.2.0-670.gadc19233.el9.x 8.1 MB/s | 305 kB 00:00 2026-02-18T06:19:00.476 INFO:teuthology.orchestra.run.trial130.stdout:(13/37): ceph-common-20.2.0-670.gadc19233.el9.x 24 MB/s | 24 MB 00:00 2026-02-18T06:19:00.476 INFO:teuthology.orchestra.run.trial130.stdout:(14/37): python3-rgw-20.2.0-670.gadc19233.el9.x 1.9 MB/s | 99 kB 00:00 2026-02-18T06:19:00.502 INFO:teuthology.orchestra.run.trial130.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 1.2 MB/s | 33 kB 00:00 2026-02-18T06:19:00.521 INFO:teuthology.orchestra.run.trial130.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 4.9 MB/s | 93 kB 00:00 2026-02-18T06:19:00.538 INFO:teuthology.orchestra.run.trial130.stdout:(17/37): nvme-cli-2.13-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2026-02-18T06:19:00.660 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:00.662 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:00.662 INFO:teuthology.orchestra.run.trial031.stdout: Package Arch Version Repository Size 2026-02-18T06:19:00.662 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:00.662 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:00.662 INFO:teuthology.orchestra.run.trial031.stdout: ceph-radosgw x86_64 2:20.2.0-670.gadc19233.el9 ceph 24 M 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout:Upgrading: 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: librados2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 3.5 M 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: librbd1 x86_64 2:20.2.0-670.gadc19233.el9 ceph 2.8 M 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout:Installing dependencies: 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: ceph-base x86_64 2:20.2.0-670.gadc19233.el9 ceph 5.9 M 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: ceph-common x86_64 2:20.2.0-670.gadc19233.el9 ceph 24 M 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: ceph-selinux x86_64 2:20.2.0-670.gadc19233.el9 ceph 25 k 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-18T06:19:00.663 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 869 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: libradosstriper1 x86_64 2:20.2.0-670.gadc19233.el9 ceph 250 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: librgw2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 6.4 M 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-18T06:19:00.664 INFO:teuthology.orchestra.run.trial031.stdout: python3-ceph-argparse x86_64 2:20.2.0-670.gadc19233.el9 ceph 45 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: python3-ceph-common x86_64 2:20.2.0-670.gadc19233.el9 ceph 175 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: python3-cephfs x86_64 2:20.2.0-670.gadc19233.el9 ceph 162 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: python3-rados x86_64 2:20.2.0-670.gadc19233.el9 ceph 323 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: python3-rbd x86_64 2:20.2.0-670.gadc19233.el9 ceph 305 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: python3-rgw x86_64 2:20.2.0-670.gadc19233.el9 ceph 99 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout:Installing weak dependencies: 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs-daemon x86_64 2:20.2.0-670.gadc19233.el9 ceph 35 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs-proxy2 x86_64 2:20.2.0-670.gadc19233.el9 ceph 24 k 2026-02-18T06:19:00.665 INFO:teuthology.orchestra.run.trial031.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-18T06:19:00.666 INFO:teuthology.orchestra.run.trial031.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-18T06:19:00.666 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:00.666 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:00.666 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:00.666 INFO:teuthology.orchestra.run.trial031.stdout:Install 35 Packages 2026-02-18T06:19:00.666 INFO:teuthology.orchestra.run.trial031.stdout:Upgrade 2 Packages 2026-02-18T06:19:00.666 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:00.667 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 79 M 2026-02-18T06:19:00.667 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:00.688 INFO:teuthology.orchestra.run.trial130.stdout:(18/37): ceph-radosgw-20.2.0-670.gadc19233.el9. 20 MB/s | 24 MB 00:01 2026-02-18T06:19:00.934 INFO:teuthology.orchestra.run.trial130.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 252 kB/s | 104 kB 00:00 2026-02-18T06:19:00.992 INFO:teuthology.orchestra.run.trial130.stdout:(20/37): libnbd-1.20.3-4.el9.x86_64.rpm 362 kB/s | 164 kB 00:00 2026-02-18T06:19:01.036 INFO:teuthology.orchestra.run.trial130.stdout:(21/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 442 kB/s | 45 kB 00:00 2026-02-18T06:19:01.141 INFO:teuthology.orchestra.run.trial130.stdout:(22/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 353 kB/s | 160 kB 00:00 2026-02-18T06:19:01.264 INFO:teuthology.orchestra.run.trial130.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2026-02-18T06:19:01.265 INFO:teuthology.orchestra.run.trial130.stdout:(24/37): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2026-02-18T06:19:01.327 INFO:teuthology.orchestra.run.trial130.stdout:(25/37): qatlib-25.08.0-2.el9.x86_64.rpm 1.3 MB/s | 240 kB 00:00 2026-02-18T06:19:01.347 INFO:teuthology.orchestra.run.trial130.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 807 kB/s | 66 kB 00:00 2026-02-18T06:19:01.348 INFO:teuthology.orchestra.run.trial130.stdout:(27/37): qatlib-service-25.08.0-2.el9.x86_64.rp 440 kB/s | 37 kB 00:00 2026-02-18T06:19:01.527 INFO:teuthology.orchestra.run.trial130.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 139 kB/s | 25 kB 00:00 2026-02-18T06:19:01.612 INFO:teuthology.orchestra.run.trial130.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 570 kB/s | 49 kB 00:00 2026-02-18T06:19:01.652 INFO:teuthology.orchestra.run.trial130.stdout:(30/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 946 kB/s | 308 kB 00:00 2026-02-18T06:19:01.675 INFO:teuthology.orchestra.run.trial130.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2026-02-18T06:19:01.786 INFO:teuthology.orchestra.run.trial130.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 1.7 MB/s | 191 kB 00:00 2026-02-18T06:19:01.791 INFO:teuthology.orchestra.run.trial130.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 5.9 MB/s | 838 kB 00:00 2026-02-18T06:19:01.897 INFO:teuthology.orchestra.run.trial031.stdout:(1/37): ceph-base-20.2.0-670.gadc19233.el9.x86_ 8.2 MB/s | 5.9 MB 00:00 2026-02-18T06:19:01.943 INFO:teuthology.orchestra.run.trial031.stdout:(2/37): ceph-selinux-20.2.0-670.gadc19233.el9.x 550 kB/s | 25 kB 00:00 2026-02-18T06:19:01.990 INFO:teuthology.orchestra.run.trial031.stdout:(3/37): libcephfs-daemon-20.2.0-670.gadc19233.e 747 kB/s | 35 kB 00:00 2026-02-18T06:19:02.031 INFO:teuthology.orchestra.run.trial130.stdout:(34/37): libarrow-9.0.0-15.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2026-02-18T06:19:02.040 INFO:teuthology.orchestra.run.trial031.stdout:(4/37): libcephfs-proxy2-20.2.0-670.gadc19233.e 480 kB/s | 24 kB 00:00 2026-02-18T06:19:02.074 INFO:teuthology.orchestra.run.trial130.stdout:(35/37): thrift-0.15.0-4.el9.x86_64.rpm 5.5 MB/s | 1.6 MB 00:00 2026-02-18T06:19:02.111 INFO:teuthology.orchestra.run.trial031.stdout:(5/37): ceph-common-20.2.0-670.gadc19233.el9.x8 26 MB/s | 24 MB 00:00 2026-02-18T06:19:02.154 INFO:teuthology.orchestra.run.trial031.stdout:(6/37): libcephfs2-20.2.0-670.gadc19233.el9.x86 7.5 MB/s | 869 kB 00:00 2026-02-18T06:19:02.157 INFO:teuthology.orchestra.run.trial031.stdout:(7/37): libradosstriper1-20.2.0-670.gadc19233.e 5.3 MB/s | 250 kB 00:00 2026-02-18T06:19:02.173 INFO:teuthology.orchestra.run.trial130.stdout:(36/37): librados2-20.2.0-670.gadc19233.el9.x86 9.2 MB/s | 3.5 MB 00:00 2026-02-18T06:19:02.202 INFO:teuthology.orchestra.run.trial031.stdout:(8/37): python3-ceph-argparse-20.2.0-670.gadc19 1.0 MB/s | 45 kB 00:00 2026-02-18T06:19:02.241 INFO:teuthology.orchestra.run.trial031.stdout:(9/37): python3-ceph-common-20.2.0-670.gadc1923 4.4 MB/s | 175 kB 00:00 2026-02-18T06:19:02.283 INFO:teuthology.orchestra.run.trial031.stdout:(10/37): python3-cephfs-20.2.0-670.gadc19233.el 3.8 MB/s | 162 kB 00:00 2026-02-18T06:19:02.333 INFO:teuthology.orchestra.run.trial031.stdout:(11/37): python3-rados-20.2.0-670.gadc19233.el9 6.3 MB/s | 323 kB 00:00 2026-02-18T06:19:02.383 INFO:teuthology.orchestra.run.trial031.stdout:(12/37): ceph-radosgw-20.2.0-670.gadc19233.el9. 20 MB/s | 24 MB 00:01 2026-02-18T06:19:02.384 INFO:teuthology.orchestra.run.trial031.stdout:(13/37): python3-rbd-20.2.0-670.gadc19233.el9.x 5.8 MB/s | 305 kB 00:00 2026-02-18T06:19:02.394 INFO:teuthology.orchestra.run.trial130.stdout:(37/37): librbd1-20.2.0-670.gadc19233.el9.x86_6 7.8 MB/s | 2.8 MB 00:00 2026-02-18T06:19:02.395 INFO:teuthology.orchestra.run.trial031.stdout:(14/37): python3-rgw-20.2.0-670.gadc19233.el9.x 8.2 MB/s | 99 kB 00:00 2026-02-18T06:19:02.398 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:02.399 INFO:teuthology.orchestra.run.trial130.stdout:Total 25 MB/s | 79 MB 00:03 2026-02-18T06:19:02.436 INFO:teuthology.orchestra.run.trial031.stdout:(15/37): librgw2-20.2.0-670.gadc19233.el9.x86_6 23 MB/s | 6.4 MB 00:00 2026-02-18T06:19:02.442 INFO:teuthology.orchestra.run.trial031.stdout:(16/37): mailcap-2.1.49-5.el9.noarch.rpm 577 kB/s | 33 kB 00:00 2026-02-18T06:19:02.482 INFO:teuthology.orchestra.run.trial031.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 2.6 MB/s | 104 kB 00:00 2026-02-18T06:19:02.499 INFO:teuthology.orchestra.run.trial031.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 9.5 MB/s | 164 kB 00:00 2026-02-18T06:19:02.503 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:02.510 INFO:teuthology.orchestra.run.trial031.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 14 MB/s | 160 kB 00:00 2026-02-18T06:19:02.519 INFO:teuthology.orchestra.run.trial031.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 5.2 MB/s | 45 kB 00:00 2026-02-18T06:19:02.521 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:02.521 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:02.522 INFO:teuthology.orchestra.run.trial031.stdout:(21/37): pciutils-3.7.0-7.el9.x86_64.rpm 1.1 MB/s | 93 kB 00:00 2026-02-18T06:19:02.539 INFO:teuthology.orchestra.run.trial031.stdout:(22/37): librdkafka-1.6.1-102.el9.x86_64.rpm 32 MB/s | 662 kB 00:00 2026-02-18T06:19:02.550 INFO:teuthology.orchestra.run.trial031.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 23 MB/s | 240 kB 00:00 2026-02-18T06:19:02.559 INFO:teuthology.orchestra.run.trial031.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 4.2 MB/s | 37 kB 00:00 2026-02-18T06:19:02.568 INFO:teuthology.orchestra.run.trial031.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 7.3 MB/s | 66 kB 00:00 2026-02-18T06:19:02.569 INFO:teuthology.orchestra.run.trial031.stdout:(26/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.9 MB/s | 292 kB 00:00 2026-02-18T06:19:02.661 INFO:teuthology.orchestra.run.trial031.stdout:(27/37): nvme-cli-2.13-1.el9.x86_64.rpm 4.1 MB/s | 1.1 MB 00:00 2026-02-18T06:19:02.687 INFO:teuthology.orchestra.run.trial031.stdout:(28/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.5 MB/s | 308 kB 00:00 2026-02-18T06:19:02.717 INFO:teuthology.orchestra.run.trial031.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.6 MB/s | 49 kB 00:00 2026-02-18T06:19:02.719 INFO:teuthology.orchestra.run.trial031.stdout:(30/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 428 kB/s | 25 kB 00:00 2026-02-18T06:19:02.746 INFO:teuthology.orchestra.run.trial031.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.3 MB/s | 67 kB 00:00 2026-02-18T06:19:02.763 INFO:teuthology.orchestra.run.trial031.stdout:(32/37): libarrow-9.0.0-15.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2026-02-18T06:19:02.780 INFO:teuthology.orchestra.run.trial031.stdout:(33/37): re2-20211101-20.el9.x86_64.rpm 5.5 MB/s | 191 kB 00:00 2026-02-18T06:19:02.813 INFO:teuthology.orchestra.run.trial031.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 32 MB/s | 1.6 MB 00:00 2026-02-18T06:19:02.818 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:02.818 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:02.835 INFO:teuthology.orchestra.run.trial031.stdout:(35/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2026-02-18T06:19:03.072 INFO:teuthology.orchestra.run.trial031.stdout:(36/37): librbd1-20.2.0-670.gadc19233.el9.x86_6 11 MB/s | 2.8 MB 00:00 2026-02-18T06:19:03.158 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:03.170 INFO:teuthology.orchestra.run.trial130.stdout: Installing : thrift-0.15.0-4.el9.x86_64 1/39 2026-02-18T06:19:03.210 INFO:teuthology.orchestra.run.trial031.stdout:(37/37): librados2-20.2.0-670.gadc19233.el9.x86 8.2 MB/s | 3.5 MB 00:00 2026-02-18T06:19:03.212 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:03.213 INFO:teuthology.orchestra.run.trial031.stdout:Total 31 MB/s | 79 MB 00:02 2026-02-18T06:19:03.309 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:03.324 INFO:teuthology.orchestra.run.trial130.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 2/39 2026-02-18T06:19:03.328 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:03.328 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:03.328 INFO:teuthology.orchestra.run.trial130.stdout: Upgrading : librados2-2:20.2.0-670.gadc19233.el9.x86_64 3/39 2026-02-18T06:19:03.376 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: librados2-2:20.2.0-670.gadc19233.el9.x86_64 3/39 2026-02-18T06:19:03.377 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 4/39 2026-02-18T06:19:03.389 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 4/39 2026-02-18T06:19:03.392 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libcephfs-daemon-2:20.2.0-670.gadc19233.el9.x86_64 5/39 2026-02-18T06:19:03.392 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 6/39 2026-02-18T06:19:03.419 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 6/39 2026-02-18T06:19:03.421 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 7/39 2026-02-18T06:19:03.428 INFO:teuthology.orchestra.run.trial130.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/39 2026-02-18T06:19:03.431 INFO:teuthology.orchestra.run.trial130.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/39 2026-02-18T06:19:03.434 INFO:teuthology.orchestra.run.trial130.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 10/39 2026-02-18T06:19:03.438 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 11/39 2026-02-18T06:19:03.448 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-ceph-argparse-2:20.2.0-670.gadc19233.el9.x 12/39 2026-02-18T06:19:03.460 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 13/39 2026-02-18T06:19:03.461 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 14/39 2026-02-18T06:19:03.473 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 14/39 2026-02-18T06:19:03.503 INFO:teuthology.orchestra.run.trial130.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-18T06:19:03.507 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-18T06:19:03.513 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-18T06:19:03.515 INFO:teuthology.orchestra.run.trial130.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-18T06:19:03.519 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-18T06:19:03.626 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:03.626 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:03.642 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-18T06:19:03.643 INFO:teuthology.orchestra.run.trial130.stdout: Upgrading : librbd1-2:20.2.0-670.gadc19233.el9.x86_64 21/39 2026-02-18T06:19:03.668 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: librbd1-2:20.2.0-670.gadc19233.el9.x86_64 21/39 2026-02-18T06:19:03.670 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 22/39 2026-02-18T06:19:03.676 INFO:teuthology.orchestra.run.trial130.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-18T06:19:03.957 INFO:teuthology.orchestra.run.trial130.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-18T06:19:03.958 INFO:teuthology.orchestra.run.trial130.stdout: Installing : librgw2-2:20.2.0-670.gadc19233.el9.x86_64 25/39 2026-02-18T06:19:03.965 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:03.974 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: librgw2-2:20.2.0-670.gadc19233.el9.x86_64 25/39 2026-02-18T06:19:03.976 INFO:teuthology.orchestra.run.trial031.stdout: Installing : thrift-0.15.0-4.el9.x86_64 1/39 2026-02-18T06:19:03.977 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 26/39 2026-02-18T06:19:03.982 INFO:teuthology.orchestra.run.trial130.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-18T06:19:03.985 INFO:teuthology.orchestra.run.trial130.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-18T06:19:03.985 INFO:teuthology.orchestra.run.trial130.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-18T06:19:04.004 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-18T06:19:04.117 INFO:teuthology.orchestra.run.trial130.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-18T06:19:04.120 INFO:teuthology.orchestra.run.trial130.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-18T06:19:04.130 INFO:teuthology.orchestra.run.trial031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 2/39 2026-02-18T06:19:04.133 INFO:teuthology.orchestra.run.trial031.stdout: Upgrading : librados2-2:20.2.0-670.gadc19233.el9.x86_64 3/39 2026-02-18T06:19:04.183 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: librados2-2:20.2.0-670.gadc19233.el9.x86_64 3/39 2026-02-18T06:19:04.184 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 4/39 2026-02-18T06:19:04.196 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 4/39 2026-02-18T06:19:04.198 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libcephfs-daemon-2:20.2.0-670.gadc19233.el9.x86_64 5/39 2026-02-18T06:19:04.199 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 6/39 2026-02-18T06:19:04.225 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 6/39 2026-02-18T06:19:04.227 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 7/39 2026-02-18T06:19:04.235 INFO:teuthology.orchestra.run.trial031.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/39 2026-02-18T06:19:04.237 INFO:teuthology.orchestra.run.trial031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/39 2026-02-18T06:19:04.240 INFO:teuthology.orchestra.run.trial031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 10/39 2026-02-18T06:19:04.245 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 11/39 2026-02-18T06:19:04.254 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-ceph-argparse-2:20.2.0-670.gadc19233.el9.x 12/39 2026-02-18T06:19:04.265 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 13/39 2026-02-18T06:19:04.266 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 14/39 2026-02-18T06:19:04.280 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 14/39 2026-02-18T06:19:04.316 INFO:teuthology.orchestra.run.trial031.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-18T06:19:04.320 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-18T06:19:04.326 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-18T06:19:04.329 INFO:teuthology.orchestra.run.trial031.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-18T06:19:04.333 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-18T06:19:04.385 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-18T06:19:04.385 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-18T06:19:04.385 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:04.427 INFO:teuthology.orchestra.run.trial130.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-18T06:19:04.429 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-ceph-common-2:20.2.0-670.gadc19233.el9.x86 33/39 2026-02-18T06:19:04.455 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-18T06:19:04.456 INFO:teuthology.orchestra.run.trial031.stdout: Upgrading : librbd1-2:20.2.0-670.gadc19233.el9.x86_64 21/39 2026-02-18T06:19:04.484 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: librbd1-2:20.2.0-670.gadc19233.el9.x86_64 21/39 2026-02-18T06:19:04.487 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 22/39 2026-02-18T06:19:04.494 INFO:teuthology.orchestra.run.trial031.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-18T06:19:04.776 INFO:teuthology.orchestra.run.trial031.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-18T06:19:04.778 INFO:teuthology.orchestra.run.trial031.stdout: Installing : librgw2-2:20.2.0-670.gadc19233.el9.x86_64 25/39 2026-02-18T06:19:04.795 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: librgw2-2:20.2.0-670.gadc19233.el9.x86_64 25/39 2026-02-18T06:19:04.800 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 26/39 2026-02-18T06:19:04.807 INFO:teuthology.orchestra.run.trial031.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-18T06:19:04.809 INFO:teuthology.orchestra.run.trial031.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-18T06:19:04.810 INFO:teuthology.orchestra.run.trial031.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-18T06:19:04.829 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-18T06:19:04.941 INFO:teuthology.orchestra.run.trial031.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-18T06:19:04.943 INFO:teuthology.orchestra.run.trial031.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-18T06:19:05.220 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-18T06:19:05.220 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-18T06:19:05.220 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:05.263 INFO:teuthology.orchestra.run.trial031.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-18T06:19:05.265 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-ceph-common-2:20.2.0-670.gadc19233.el9.x86 33/39 2026-02-18T06:19:05.509 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 34/39 2026-02-18T06:19:05.512 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 34/39 2026-02-18T06:19:05.808 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 34/39 2026-02-18T06:19:05.812 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 35/39 2026-02-18T06:19:05.854 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 35/39 2026-02-18T06:19:05.854 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-18T06:19:05.854 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-18T06:19:05.854 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:05.858 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 36/39 2026-02-18T06:19:06.342 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 34/39 2026-02-18T06:19:06.344 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 34/39 2026-02-18T06:19:06.638 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 34/39 2026-02-18T06:19:06.641 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 35/39 2026-02-18T06:19:06.684 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 35/39 2026-02-18T06:19:06.684 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-18T06:19:06.684 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-18T06:19:06.684 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:06.687 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 36/39 2026-02-18T06:19:10.432 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 36/39 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout:skipping the directory /sys 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout:skipping the directory /proc 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout:skipping the directory /mnt 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout:skipping the directory /var/tmp 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout:skipping the directory /home 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout:skipping the directory /root 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout:skipping the directory /tmp 2026-02-18T06:19:10.433 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:11.272 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 36/39 2026-02-18T06:19:11.272 INFO:teuthology.orchestra.run.trial031.stdout:skipping the directory /sys 2026-02-18T06:19:11.272 INFO:teuthology.orchestra.run.trial031.stdout:skipping the directory /proc 2026-02-18T06:19:11.272 INFO:teuthology.orchestra.run.trial031.stdout:skipping the directory /mnt 2026-02-18T06:19:11.272 INFO:teuthology.orchestra.run.trial031.stdout:skipping the directory /var/tmp 2026-02-18T06:19:11.272 INFO:teuthology.orchestra.run.trial031.stdout:skipping the directory /home 2026-02-18T06:19:11.273 INFO:teuthology.orchestra.run.trial031.stdout:skipping the directory /root 2026-02-18T06:19:11.273 INFO:teuthology.orchestra.run.trial031.stdout:skipping the directory /tmp 2026-02-18T06:19:11.273 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:11.463 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 37/39 2026-02-18T06:19:11.487 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 37/39 2026-02-18T06:19:11.488 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:11.488 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-18T06:19:11.488 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-18T06:19:11.488 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-18T06:19:11.488 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:11.490 INFO:teuthology.orchestra.run.trial130.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-18T06:19:11.504 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-18T06:19:11.504 INFO:teuthology.orchestra.run.trial130.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-18T06:19:11.559 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-18T06:19:11.559 INFO:teuthology.orchestra.run.trial130.stdout:Creating group 'qat' with GID 992. 2026-02-18T06:19:11.559 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:11.868 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 1/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 2/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 3/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 4/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libcephfs-daemon-2:20.2.0-670.gadc19233.el9.x86_64 5/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 6/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 7/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 8/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librgw2-2:20.2.0-670.gadc19233.el9.x86_64 9/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-ceph-argparse-2:20.2.0-670.gadc19233.el9.x 10/39 2026-02-18T06:19:11.869 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-ceph-common-2:20.2.0-670.gadc19233.el9.x86 11/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 12/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 13/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 14/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 15/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-18T06:19:11.870 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-18T06:19:11.871 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-18T06:19:11.872 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-18T06:19:11.872 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librados2-2:20.2.0-670.gadc19233.el9.x86_64 36/39 2026-02-18T06:19:11.872 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-18T06:19:11.872 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librbd1-2:20.2.0-670.gadc19233.el9.x86_64 38/39 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout:Upgraded: 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: librados2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: librbd1-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.925 INFO:teuthology.orchestra.run.trial130.stdout: ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs-daemon-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-18T06:19:11.926 INFO:teuthology.orchestra.run.trial130.stdout: libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: librgw2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: python3-ceph-argparse-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: python3-ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.927 INFO:teuthology.orchestra.run.trial130.stdout: python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout: python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout: re2-1:20211101-20.el9.x86_64 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:11.928 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:11.968 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-test 2026-02-18T06:19:12.280 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:14 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:12.302 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 37/39 2026-02-18T06:19:12.325 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 37/39 2026-02-18T06:19:12.325 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:12.325 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-18T06:19:12.326 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-18T06:19:12.326 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-18T06:19:12.326 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:12.327 INFO:teuthology.orchestra.run.trial031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-18T06:19:12.339 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-18T06:19:12.339 INFO:teuthology.orchestra.run.trial031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-18T06:19:12.361 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout: Package Arch Version Repository Size 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout: ceph-test x86_64 2:20.2.0-670.gadc19233.el9 ceph 84 M 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout:Installing dependencies: 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout: libcephsqlite x86_64 2:20.2.0-670.gadc19233.el9 ceph 164 k 2026-02-18T06:19:12.362 INFO:teuthology.orchestra.run.trial130.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-18T06:19:12.363 INFO:teuthology.orchestra.run.trial130.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-18T06:19:12.363 INFO:teuthology.orchestra.run.trial130.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-18T06:19:12.363 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:12.363 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:12.363 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:12.363 INFO:teuthology.orchestra.run.trial130.stdout:Install 5 Packages 2026-02-18T06:19:12.363 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:12.364 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 85 M 2026-02-18T06:19:12.364 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 364 M 2026-02-18T06:19:12.364 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:12.392 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-18T06:19:12.392 INFO:teuthology.orchestra.run.trial031.stdout:Creating group 'qat' with GID 992. 2026-02-18T06:19:12.392 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:12.502 INFO:teuthology.orchestra.run.trial130.stdout:(1/5): libxslt-1.1.34-12.el9.x86_64.rpm 4.9 MB/s | 233 kB 00:00 2026-02-18T06:19:12.521 INFO:teuthology.orchestra.run.trial130.stdout:(2/5): socat-1.7.4.1-8.el9.x86_64.rpm 16 MB/s | 303 kB 00:00 2026-02-18T06:19:12.532 INFO:teuthology.orchestra.run.trial130.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 5.7 MB/s | 64 kB 00:00 2026-02-18T06:19:12.542 INFO:teuthology.orchestra.run.trial130.stdout:(4/5): libcephsqlite-20.2.0-670.gadc19233.el9.x 1.9 MB/s | 164 kB 00:00 2026-02-18T06:19:12.703 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 1/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 2/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 3/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 4/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libcephfs-daemon-2:20.2.0-670.gadc19233.el9.x86_64 5/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 6/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 7/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 8/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librgw2-2:20.2.0-670.gadc19233.el9.x86_64 9/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-ceph-argparse-2:20.2.0-670.gadc19233.el9.x 10/39 2026-02-18T06:19:12.704 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-ceph-common-2:20.2.0-670.gadc19233.el9.x86 11/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 12/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 13/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 14/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 15/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-18T06:19:12.705 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librados2-2:20.2.0-670.gadc19233.el9.x86_64 36/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-18T06:19:12.706 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librbd1-2:20.2.0-670.gadc19233.el9.x86_64 38/39 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout:Upgraded: 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout: librados2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout: librbd1-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout: ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.761 INFO:teuthology.orchestra.run.trial031.stdout: ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: ceph-radosgw-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: ceph-selinux-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs-daemon-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs-proxy2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-18T06:19:12.762 INFO:teuthology.orchestra.run.trial031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: libradosstriper1-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: librgw2-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: python3-ceph-argparse-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: python3-ceph-common-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.763 INFO:teuthology.orchestra.run.trial031.stdout: python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: re2-1:20211101-20.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:12.764 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:12.803 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-test 2026-02-18T06:19:13.118 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:14 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:13.200 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:13.200 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:13.200 INFO:teuthology.orchestra.run.trial031.stdout: Package Arch Version Repository Size 2026-02-18T06:19:13.200 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:13.200 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:13.200 INFO:teuthology.orchestra.run.trial031.stdout: ceph-test x86_64 2:20.2.0-670.gadc19233.el9 ceph 84 M 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout:Installing dependencies: 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout: libcephsqlite x86_64 2:20.2.0-670.gadc19233.el9 ceph 164 k 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout:Install 5 Packages 2026-02-18T06:19:13.201 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:13.202 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 85 M 2026-02-18T06:19:13.202 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 364 M 2026-02-18T06:19:13.202 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:13.428 INFO:teuthology.orchestra.run.trial031.stdout:(1/5): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-18T06:19:13.480 INFO:teuthology.orchestra.run.trial031.stdout:(2/5): socat-1.7.4.1-8.el9.x86_64.rpm 5.6 MB/s | 303 kB 00:00 2026-02-18T06:19:13.506 INFO:teuthology.orchestra.run.trial031.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 64 kB 00:00 2026-02-18T06:19:13.709 INFO:teuthology.orchestra.run.trial031.stdout:(4/5): libcephsqlite-20.2.0-670.gadc19233.el9.x 369 kB/s | 164 kB 00:00 2026-02-18T06:19:14.620 INFO:teuthology.orchestra.run.trial130.stdout:(5/5): ceph-test-20.2.0-670.gadc19233.el9.x86_6 39 MB/s | 84 MB 00:02 2026-02-18T06:19:14.621 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:14.622 INFO:teuthology.orchestra.run.trial130.stdout:Total 37 MB/s | 85 MB 00:02 2026-02-18T06:19:14.647 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:14.657 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:14.658 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:14.943 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:14.944 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:15.250 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:15.256 INFO:teuthology.orchestra.run.trial130.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-18T06:19:15.260 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-18T06:19:15.268 INFO:teuthology.orchestra.run.trial130.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-18T06:19:15.269 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 4/5 2026-02-18T06:19:18.968 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 4/5 2026-02-18T06:19:18.986 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 5/5 2026-02-18T06:19:19.081 INFO:teuthology.orchestra.run.trial031.stdout:(5/5): ceph-test-20.2.0-670.gadc19233.el9.x86_6 14 MB/s | 84 MB 00:05 2026-02-18T06:19:19.082 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:19.082 INFO:teuthology.orchestra.run.trial031.stdout:Total 14 MB/s | 85 MB 00:05 2026-02-18T06:19:19.109 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:19.119 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:19.119 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:19.357 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 5/5 2026-02-18T06:19:19.357 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 1/5 2026-02-18T06:19:19.358 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 2/5 2026-02-18T06:19:19.358 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-18T06:19:19.358 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-18T06:19:19.406 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:19.407 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout: ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout: libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-18T06:19:19.410 INFO:teuthology.orchestra.run.trial130.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-18T06:19:19.411 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:19.411 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:19.447 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph 2026-02-18T06:19:19.715 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:19.721 INFO:teuthology.orchestra.run.trial031.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-18T06:19:19.725 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-18T06:19:19.733 INFO:teuthology.orchestra.run.trial031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-18T06:19:19.733 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 4/5 2026-02-18T06:19:19.753 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:21 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:19.838 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:19.841 INFO:teuthology.orchestra.run.trial130.stdout:====================================================================================== 2026-02-18T06:19:19.841 INFO:teuthology.orchestra.run.trial130.stdout: Package Arch Version Repository Size 2026-02-18T06:19:19.841 INFO:teuthology.orchestra.run.trial130.stdout:====================================================================================== 2026-02-18T06:19:19.841 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: ceph x86_64 2:20.2.0-670.gadc19233.el9 ceph 6.5 k 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout:Installing dependencies: 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: ceph-grafana-dashboards noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 43 k 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mds x86_64 2:20.2.0-670.gadc19233.el9 ceph 2.3 M 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr x86_64 2:20.2.0-670.gadc19233.el9 ceph 961 k 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-modules-core noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 290 k 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mon x86_64 2:20.2.0-670.gadc19233.el9 ceph 5.1 M 2026-02-18T06:19:19.842 INFO:teuthology.orchestra.run.trial130.stdout: ceph-osd x86_64 2:20.2.0-670.gadc19233.el9 ceph 17 M 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: ceph-prometheus-alerts noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 17 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: cephadm noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 1.0 M 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-18T06:19:19.843 INFO:teuthology.orchestra.run.trial130.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-18T06:19:19.844 INFO:teuthology.orchestra.run.trial130.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-18T06:19:19.845 INFO:teuthology.orchestra.run.trial130.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-18T06:19:19.846 INFO:teuthology.orchestra.run.trial130.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-18T06:19:19.847 INFO:teuthology.orchestra.run.trial130.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-18T06:19:19.848 INFO:teuthology.orchestra.run.trial130.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout:Installing weak dependencies: 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-cephadm noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 173 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-dashboard noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 11 M 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 7.4 M 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-k8sevents noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 22 k 2026-02-18T06:19:19.849 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-rook noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 50 k 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout: ceph-volume noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 297 k 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout:====================================================================================== 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout:Install 95 Packages 2026-02-18T06:19:19.850 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:19.851 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 101 M 2026-02-18T06:19:19.851 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 561 M 2026-02-18T06:19:19.851 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:20.441 INFO:teuthology.orchestra.run.trial130.stdout:(1/95): ceph-20.2.0-670.gadc19233.el9.x86_64.rp 129 kB/s | 6.5 kB 00:00 2026-02-18T06:19:20.530 INFO:teuthology.orchestra.run.trial130.stdout:(2/95): ceph-mgr-20.2.0-670.gadc19233.el9.x86_6 6.8 MB/s | 961 kB 00:00 2026-02-18T06:19:20.556 INFO:teuthology.orchestra.run.trial130.stdout:(3/95): ceph-mds-20.2.0-670.gadc19233.el9.x86_6 14 MB/s | 2.3 MB 00:00 2026-02-18T06:19:20.593 INFO:teuthology.orchestra.run.trial130.stdout:(4/95): ceph-grafana-dashboards-20.2.0-670.gadc 1.1 MB/s | 43 kB 00:00 2026-02-18T06:19:20.633 INFO:teuthology.orchestra.run.trial130.stdout:(5/95): ceph-mgr-cephadm-20.2.0-670.gadc19233.e 4.3 MB/s | 173 kB 00:00 2026-02-18T06:19:20.716 INFO:teuthology.orchestra.run.trial130.stdout:(6/95): ceph-mon-20.2.0-670.gadc19233.el9.x86_6 18 MB/s | 5.1 MB 00:00 2026-02-18T06:19:21.160 INFO:teuthology.orchestra.run.trial130.stdout:(7/95): ceph-mgr-diskprediction-local-20.2.0-67 17 MB/s | 7.4 MB 00:00 2026-02-18T06:19:21.195 INFO:teuthology.orchestra.run.trial130.stdout:(8/95): ceph-mgr-k8sevents-20.2.0-670.gadc19233 630 kB/s | 22 kB 00:00 2026-02-18T06:19:21.217 INFO:teuthology.orchestra.run.trial130.stdout:(9/95): ceph-mgr-dashboard-20.2.0-670.gadc19233 18 MB/s | 11 MB 00:00 2026-02-18T06:19:21.240 INFO:teuthology.orchestra.run.trial130.stdout:(10/95): ceph-osd-20.2.0-670.gadc19233.el9.x86_ 24 MB/s | 17 MB 00:00 2026-02-18T06:19:21.241 INFO:teuthology.orchestra.run.trial130.stdout:(11/95): ceph-mgr-rook-20.2.0-670.gadc19233.el9 2.0 MB/s | 50 kB 00:00 2026-02-18T06:19:21.242 INFO:teuthology.orchestra.run.trial130.stdout:(12/95): ceph-mgr-modules-core-20.2.0-670.gadc1 6.1 MB/s | 290 kB 00:00 2026-02-18T06:19:21.252 INFO:teuthology.orchestra.run.trial130.stdout:(13/95): ceph-prometheus-alerts-20.2.0-670.gadc 1.5 MB/s | 17 kB 00:00 2026-02-18T06:19:21.253 INFO:teuthology.orchestra.run.trial130.stdout:(14/95): ceph-volume-20.2.0-670.gadc19233.el9.n 24 MB/s | 297 kB 00:00 2026-02-18T06:19:21.266 INFO:teuthology.orchestra.run.trial130.stdout:(15/95): cephadm-20.2.0-670.gadc19233.el9.noarc 42 MB/s | 1.0 MB 00:00 2026-02-18T06:19:21.943 INFO:teuthology.orchestra.run.trial130.stdout:(16/95): libconfig-1.7.2-9.el9.x86_64.rpm 106 kB/s | 72 kB 00:00 2026-02-18T06:19:22.086 INFO:teuthology.orchestra.run.trial130.stdout:(17/95): cryptsetup-2.8.1-2.el9.x86_64.rpm 420 kB/s | 350 kB 00:00 2026-02-18T06:19:22.143 INFO:teuthology.orchestra.run.trial130.stdout:(18/95): ledmon-libs-1.1.0-3.el9.x86_64.rpm 45 kB/s | 40 kB 00:00 2026-02-18T06:19:22.244 INFO:teuthology.orchestra.run.trial130.stdout:(19/95): libquadmath-11.5.0-14.el9.x86_64.rpm 1.1 MB/s | 184 kB 00:00 2026-02-18T06:19:22.342 INFO:teuthology.orchestra.run.trial130.stdout:(20/95): libgfortran-11.5.0-14.el9.x86_64.rpm 1.9 MB/s | 794 kB 00:00 2026-02-18T06:19:22.461 INFO:teuthology.orchestra.run.trial130.stdout:(21/95): python3-cffi-1.14.5-5.el9.x86_64.rpm 798 kB/s | 253 kB 00:00 2026-02-18T06:19:22.502 INFO:teuthology.orchestra.run.trial130.stdout:(22/95): python3-ply-3.11-14.el9.noarch.rpm 665 kB/s | 106 kB 00:00 2026-02-18T06:19:22.560 INFO:teuthology.orchestra.run.trial130.stdout:(23/95): python3-cryptography-36.0.1-5.el9.x86_ 3.9 MB/s | 1.2 MB 00:00 2026-02-18T06:19:22.620 INFO:teuthology.orchestra.run.trial130.stdout:(24/95): python3-pycparser-2.20-6.el9.noarch.rp 848 kB/s | 135 kB 00:00 2026-02-18T06:19:22.661 INFO:teuthology.orchestra.run.trial130.stdout:(25/95): python3-pyparsing-2.4.7-9.el9.noarch.r 945 kB/s | 150 kB 00:00 2026-02-18T06:19:22.720 INFO:teuthology.orchestra.run.trial130.stdout:(26/95): python3-requests-2.25.1-10.el9.noarch. 792 kB/s | 126 kB 00:00 2026-02-18T06:19:22.780 INFO:teuthology.orchestra.run.trial130.stdout:(27/95): python3-urllib3-1.26.5-7.el9.noarch.rp 1.3 MB/s | 218 kB 00:00 2026-02-18T06:19:22.820 INFO:teuthology.orchestra.run.trial130.stdout:(28/95): unzip-6.0-59.el9.x86_64.rpm 1.1 MB/s | 182 kB 00:00 2026-02-18T06:19:22.833 INFO:teuthology.orchestra.run.trial130.stdout:(29/95): flexiblas-3.0.4-9.el9.x86_64.rpm 556 kB/s | 30 kB 00:00 2026-02-18T06:19:22.876 INFO:teuthology.orchestra.run.trial130.stdout:(30/95): flexiblas-openblas-openmp-3.0.4-9.el9. 350 kB/s | 15 kB 00:00 2026-02-18T06:19:22.882 INFO:teuthology.orchestra.run.trial130.stdout:(31/95): zip-3.0-35.el9.x86_64.rpm 1.6 MB/s | 266 kB 00:00 2026-02-18T06:19:22.941 INFO:teuthology.orchestra.run.trial130.stdout:(32/95): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2026-02-18T06:19:22.945 INFO:teuthology.orchestra.run.trial130.stdout:(33/95): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 24 MB/s | 3.0 MB 00:00 2026-02-18T06:19:22.946 INFO:teuthology.orchestra.run.trial130.stdout:(34/95): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 3.4 MB/s | 246 kB 00:00 2026-02-18T06:19:22.989 INFO:teuthology.orchestra.run.trial130.stdout:(35/95): openblas-0.3.29-1.el9.x86_64.rpm 879 kB/s | 42 kB 00:00 2026-02-18T06:19:23.051 INFO:teuthology.orchestra.run.trial130.stdout:(36/95): protobuf-3.14.0-17.el9.x86_64.rpm 9.6 MB/s | 1.0 MB 00:00 2026-02-18T06:19:23.133 INFO:teuthology.orchestra.run.trial130.stdout:(37/95): python3-devel-3.9.25-2.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2026-02-18T06:19:23.160 INFO:teuthology.orchestra.run.trial130.stdout:(38/95): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 6.0 MB 00:00 2026-02-18T06:19:23.198 INFO:teuthology.orchestra.run.trial130.stdout:(39/95): openblas-openmp-0.3.29-1.el9.x86_64.rp 21 MB/s | 5.3 MB 00:00 2026-02-18T06:19:23.213 INFO:teuthology.orchestra.run.trial130.stdout:(40/95): python3-jmespath-1.0.1-1.el9.noarch.rp 894 kB/s | 48 kB 00:00 2026-02-18T06:19:23.216 INFO:teuthology.orchestra.run.trial130.stdout:(41/95): python3-jinja2-2.11.3-8.el9.noarch.rpm 2.9 MB/s | 249 kB 00:00 2026-02-18T06:19:23.257 INFO:teuthology.orchestra.run.trial130.stdout:(42/95): python3-libstoragemgmt-1.10.1-1.el9.x8 2.9 MB/s | 177 kB 00:00 2026-02-18T06:19:23.258 INFO:teuthology.orchestra.run.trial130.stdout:(43/95): python3-markupsafe-1.1.1-12.el9.x86_64 821 kB/s | 35 kB 00:00 2026-02-18T06:19:23.318 INFO:teuthology.orchestra.run.trial130.stdout:(44/95): python3-lxml-4.6.5-3.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2026-02-18T06:19:23.343 INFO:teuthology.orchestra.run.trial130.stdout:(45/95): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.1 MB/s | 442 kB 00:00 2026-02-18T06:19:23.369 INFO:teuthology.orchestra.run.trial130.stdout:(46/95): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2026-02-18T06:19:23.416 INFO:teuthology.orchestra.run.trial130.stdout:(47/95): python3-protobuf-3.14.0-17.el9.noarch. 3.6 MB/s | 267 kB 00:00 2026-02-18T06:19:23.431 INFO:teuthology.orchestra.run.trial130.stdout:(48/95): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.5 MB/s | 159 kB 00:00 2026-02-18T06:19:23.460 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 4/5 2026-02-18T06:19:23.478 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 5/5 2026-02-18T06:19:23.481 INFO:teuthology.orchestra.run.trial130.stdout:(49/95): python3-requests-oauthlib-1.3.0-12.el9 1.0 MB/s | 54 kB 00:00 2026-02-18T06:19:23.484 INFO:teuthology.orchestra.run.trial130.stdout:(50/95): python3-pyasn1-modules-0.4.8-6.el9.noa 4.0 MB/s | 279 kB 00:00 2026-02-18T06:19:23.536 INFO:teuthology.orchestra.run.trial130.stdout:(51/95): python3-numpy-1.23.5-2.el9.x86_64.rpm 22 MB/s | 6.1 MB 00:00 2026-02-18T06:19:23.537 INFO:teuthology.orchestra.run.trial130.stdout:(52/95): python3-toml-0.10.2-6.el9.noarch.rpm 795 kB/s | 42 kB 00:00 2026-02-18T06:19:23.597 INFO:teuthology.orchestra.run.trial130.stdout:(53/95): xmlsec1-1.2.29-13.el9.x86_64.rpm 3.0 MB/s | 189 kB 00:00 2026-02-18T06:19:23.598 INFO:teuthology.orchestra.run.trial130.stdout:(54/95): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.4 MB/s | 90 kB 00:00 2026-02-18T06:19:23.608 INFO:teuthology.orchestra.run.trial130.stdout:(55/95): lua-devel-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 22 kB 00:00 2026-02-18T06:19:23.624 INFO:teuthology.orchestra.run.trial130.stdout:(56/95): protobuf-compiler-3.14.0-17.el9.x86_64 33 MB/s | 862 kB 00:00 2026-02-18T06:19:23.696 INFO:teuthology.orchestra.run.trial130.stdout:(57/95): grpc-data-1.46.7-10.el9.noarch.rpm 272 kB/s | 19 kB 00:00 2026-02-18T06:19:23.768 INFO:teuthology.orchestra.run.trial130.stdout:(58/95): luarocks-3.9.2-5.el9.noarch.rpm 2.1 MB/s | 151 kB 00:00 2026-02-18T06:19:23.776 INFO:teuthology.orchestra.run.trial130.stdout:(59/95): abseil-cpp-20211102.0-4.el9.x86_64.rpm 3.2 MB/s | 551 kB 00:00 2026-02-18T06:19:23.801 INFO:teuthology.orchestra.run.trial130.stdout:(60/95): python3-autocommand-2.2.2-8.el9.noarch 1.2 MB/s | 29 kB 00:00 2026-02-18T06:19:23.827 INFO:teuthology.orchestra.run.trial130.stdout:(61/95): python3-backports-tarfile-1.2.0-1.el9. 2.3 MB/s | 60 kB 00:00 2026-02-18T06:19:23.828 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 5/5 2026-02-18T06:19:23.828 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 1/5 2026-02-18T06:19:23.828 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 2/5 2026-02-18T06:19:23.828 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-18T06:19:23.828 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-18T06:19:23.843 INFO:teuthology.orchestra.run.trial130.stdout:(62/95): python3-asyncssh-2.13.2-5.el9.noarch.r 7.2 MB/s | 548 kB 00:00 2026-02-18T06:19:23.852 INFO:teuthology.orchestra.run.trial130.stdout:(63/95): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2026-02-18T06:19:23.869 INFO:teuthology.orchestra.run.trial130.stdout:(64/95): python3-cachetools-4.2.4-1.el9.noarch. 1.2 MB/s | 32 kB 00:00 2026-02-18T06:19:23.877 INFO:teuthology.orchestra.run.trial130.stdout:(65/95): python3-certifi-2023.05.07-4.el9.noarc 575 kB/s | 14 kB 00:00 2026-02-18T06:19:23.881 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-18T06:19:23.881 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:23.881 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:23.882 INFO:teuthology.orchestra.run.trial031.stdout: ceph-test-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:23.882 INFO:teuthology.orchestra.run.trial031.stdout: libcephsqlite-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:23.882 INFO:teuthology.orchestra.run.trial031.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-18T06:19:23.882 INFO:teuthology.orchestra.run.trial031.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-18T06:19:23.882 INFO:teuthology.orchestra.run.trial031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-18T06:19:23.882 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:23.882 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:23.896 INFO:teuthology.orchestra.run.trial130.stdout:(66/95): python3-cheroot-10.0.1-4.el9.noarch.rp 6.2 MB/s | 173 kB 00:00 2026-02-18T06:19:23.904 INFO:teuthology.orchestra.run.trial130.stdout:(67/95): python3-cherrypy-18.6.1-2.el9.noarch.r 13 MB/s | 358 kB 00:00 2026-02-18T06:19:23.922 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph 2026-02-18T06:19:23.926 INFO:teuthology.orchestra.run.trial130.stdout:(68/95): python3-google-auth-2.45.0-1.el9.noarc 8.3 MB/s | 254 kB 00:00 2026-02-18T06:19:23.954 INFO:teuthology.orchestra.run.trial130.stdout:(69/95): python3-grpcio-tools-1.46.7-10.el9.x86 5.1 MB/s | 144 kB 00:00 2026-02-18T06:19:23.983 INFO:teuthology.orchestra.run.trial130.stdout:(70/95): python3-influxdb-5.3.1-1.el9.noarch.rp 4.8 MB/s | 139 kB 00:00 2026-02-18T06:19:24.005 INFO:teuthology.orchestra.run.trial130.stdout:(71/95): python3-grpcio-1.46.7-10.el9.x86_64.rp 20 MB/s | 2.0 MB 00:00 2026-02-18T06:19:24.009 INFO:teuthology.orchestra.run.trial130.stdout:(72/95): python3-isodate-0.6.1-3.el9.noarch.rpm 2.1 MB/s | 56 kB 00:00 2026-02-18T06:19:24.044 INFO:teuthology.orchestra.run.trial130.stdout:(73/95): python3-scipy-1.9.3-2.el9.x86_64.rpm 34 MB/s | 19 MB 00:00 2026-02-18T06:19:24.044 INFO:teuthology.orchestra.run.trial130.stdout:(74/95): python3-jaraco-8.2.1-3.el9.noarch.rpm 277 kB/s | 11 kB 00:00 2026-02-18T06:19:24.045 INFO:teuthology.orchestra.run.trial130.stdout:(75/95): python3-jaraco-classes-3.2.1-5.el9.noa 493 kB/s | 18 kB 00:00 2026-02-18T06:19:24.070 INFO:teuthology.orchestra.run.trial130.stdout:(76/95): python3-jaraco-collections-3.0.0-8.el9 899 kB/s | 23 kB 00:00 2026-02-18T06:19:24.070 INFO:teuthology.orchestra.run.trial130.stdout:(77/95): python3-jaraco-context-6.0.1-3.el9.noa 762 kB/s | 20 kB 00:00 2026-02-18T06:19:24.095 INFO:teuthology.orchestra.run.trial130.stdout:(78/95): python3-jaraco-text-4.0.0-2.el9.noarch 1.0 MB/s | 26 kB 00:00 2026-02-18T06:19:24.118 INFO:teuthology.orchestra.run.trial130.stdout:(79/95): python3-jaraco-functools-3.5.0-2.el9.n 263 kB/s | 19 kB 00:00 2026-02-18T06:19:24.122 INFO:teuthology.orchestra.run.trial130.stdout:(80/95): python3-kubernetes-26.1.0-3.el9.noarch 20 MB/s | 1.0 MB 00:00 2026-02-18T06:19:24.122 INFO:teuthology.orchestra.run.trial130.stdout:(81/95): python3-more-itertools-8.12.0-2.el9.no 2.8 MB/s | 79 kB 00:00 2026-02-18T06:19:24.147 INFO:teuthology.orchestra.run.trial130.stdout:(82/95): python3-portend-3.1.0-2.el9.noarch.rpm 673 kB/s | 16 kB 00:00 2026-02-18T06:19:24.148 INFO:teuthology.orchestra.run.trial130.stdout:(83/95): python3-natsort-7.1.1-5.el9.noarch.rpm 2.1 MB/s | 58 kB 00:00 2026-02-18T06:19:24.170 INFO:teuthology.orchestra.run.trial130.stdout:(84/95): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.6 MB/s | 86 kB 00:00 2026-02-18T06:19:24.173 INFO:teuthology.orchestra.run.trial130.stdout:(85/95): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.3 MB/s | 90 kB 00:00 2026-02-18T06:19:24.174 INFO:teuthology.orchestra.run.trial130.stdout:(86/95): python3-repoze-lru-0.7-16.el9.noarch.r 1.2 MB/s | 31 kB 00:00 2026-02-18T06:19:24.200 INFO:teuthology.orchestra.run.trial130.stdout:(87/95): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2026-02-18T06:19:24.200 INFO:teuthology.orchestra.run.trial130.stdout:(88/95): python3-saml-1.16.0-1.el9.noarch.rpm 4.6 MB/s | 125 kB 00:00 2026-02-18T06:19:24.222 INFO:teuthology.orchestra.run.trial130.stdout:(89/95): python3-routes-2.5.1-5.el9.noarch.rpm 3.5 MB/s | 188 kB 00:00 2026-02-18T06:19:24.225 INFO:teuthology.orchestra.run.trial130.stdout:(90/95): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2026-02-18T06:19:24.226 INFO:teuthology.orchestra.run.trial130.stdout:(91/95): python3-typing-extensions-4.15.0-1.el9 3.2 MB/s | 86 kB 00:00 2026-02-18T06:19:24.233 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:25 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:24.251 INFO:teuthology.orchestra.run.trial130.stdout:(92/95): python3-websocket-client-1.2.3-2.el9.n 3.0 MB/s | 90 kB 00:00 2026-02-18T06:19:24.252 INFO:teuthology.orchestra.run.trial130.stdout:(93/95): python3-xmltodict-0.12.0-15.el9.noarch 868 kB/s | 22 kB 00:00 2026-02-18T06:19:24.254 INFO:teuthology.orchestra.run.trial130.stdout:(94/95): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.7 MB/s | 48 kB 00:00 2026-02-18T06:19:24.278 INFO:teuthology.orchestra.run.trial130.stdout:(95/95): python3-zc-lockfile-2.0-10.el9.noarch. 751 kB/s | 20 kB 00:00 2026-02-18T06:19:24.280 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:24.280 INFO:teuthology.orchestra.run.trial130.stdout:Total 23 MB/s | 101 MB 00:04 2026-02-18T06:19:24.318 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout:====================================================================================== 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout: Package Arch Version Repository Size 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout:====================================================================================== 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout: ceph x86_64 2:20.2.0-670.gadc19233.el9 ceph 6.5 k 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout:Installing dependencies: 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-18T06:19:24.321 INFO:teuthology.orchestra.run.trial031.stdout: ceph-grafana-dashboards noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 43 k 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mds x86_64 2:20.2.0-670.gadc19233.el9 ceph 2.3 M 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr x86_64 2:20.2.0-670.gadc19233.el9 ceph 961 k 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-modules-core noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 290 k 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mon x86_64 2:20.2.0-670.gadc19233.el9 ceph 5.1 M 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: ceph-osd x86_64 2:20.2.0-670.gadc19233.el9 ceph 17 M 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: ceph-prometheus-alerts noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 17 k 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: cephadm noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 1.0 M 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-18T06:19:24.322 INFO:teuthology.orchestra.run.trial031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-18T06:19:24.323 INFO:teuthology.orchestra.run.trial031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-18T06:19:24.324 INFO:teuthology.orchestra.run.trial031.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-18T06:19:24.325 INFO:teuthology.orchestra.run.trial031.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-18T06:19:24.326 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-18T06:19:24.327 INFO:teuthology.orchestra.run.trial031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout:Installing weak dependencies: 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-cephadm noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 173 k 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-dashboard noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 11 M 2026-02-18T06:19:24.328 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 7.4 M 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-k8sevents noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 22 k 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-rook noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 50 k 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout: ceph-volume noarch 2:20.2.0-670.gadc19233.el9 ceph-noarch 297 k 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout:====================================================================================== 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout:Install 95 Packages 2026-02-18T06:19:24.329 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:24.330 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 101 M 2026-02-18T06:19:24.330 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 561 M 2026-02-18T06:19:24.330 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:24.724 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:24.761 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:24.761 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:24.916 INFO:teuthology.orchestra.run.trial031.stdout:(1/95): ceph-20.2.0-670.gadc19233.el9.x86_64.rp 116 kB/s | 6.5 kB 00:00 2026-02-18T06:19:24.992 INFO:teuthology.orchestra.run.trial031.stdout:(2/95): ceph-mgr-20.2.0-670.gadc19233.el9.x86_6 7.2 MB/s | 961 kB 00:00 2026-02-18T06:19:25.011 INFO:teuthology.orchestra.run.trial031.stdout:(3/95): ceph-mds-20.2.0-670.gadc19233.el9.x86_6 16 MB/s | 2.3 MB 00:00 2026-02-18T06:19:25.036 INFO:teuthology.orchestra.run.trial031.stdout:(4/95): ceph-grafana-dashboards-20.2.0-670.gadc 1.7 MB/s | 43 kB 00:00 2026-02-18T06:19:25.066 INFO:teuthology.orchestra.run.trial031.stdout:(5/95): ceph-mgr-cephadm-20.2.0-670.gadc19233.e 5.7 MB/s | 173 kB 00:00 2026-02-18T06:19:25.193 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:25.193 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:25.293 INFO:teuthology.orchestra.run.trial031.stdout:(6/95): ceph-mon-20.2.0-670.gadc19233.el9.x86_6 13 MB/s | 5.1 MB 00:00 2026-02-18T06:19:25.473 INFO:teuthology.orchestra.run.trial031.stdout:(7/95): ceph-mgr-dashboard-20.2.0-670.gadc19233 26 MB/s | 11 MB 00:00 2026-02-18T06:19:25.502 INFO:teuthology.orchestra.run.trial031.stdout:(8/95): ceph-mgr-k8sevents-20.2.0-670.gadc19233 765 kB/s | 22 kB 00:00 2026-02-18T06:19:25.548 INFO:teuthology.orchestra.run.trial031.stdout:(9/95): ceph-mgr-modules-core-20.2.0-670.gadc19 6.2 MB/s | 290 kB 00:00 2026-02-18T06:19:25.591 INFO:teuthology.orchestra.run.trial031.stdout:(10/95): ceph-mgr-rook-20.2.0-670.gadc19233.el9 1.1 MB/s | 50 kB 00:00 2026-02-18T06:19:25.635 INFO:teuthology.orchestra.run.trial031.stdout:(11/95): ceph-prometheus-alerts-20.2.0-670.gadc 391 kB/s | 17 kB 00:00 2026-02-18T06:19:25.651 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:25.653 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/95 2026-02-18T06:19:25.657 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/95 2026-02-18T06:19:25.678 INFO:teuthology.orchestra.run.trial130.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/95 2026-02-18T06:19:25.693 INFO:teuthology.orchestra.run.trial031.stdout:(12/95): ceph-osd-20.2.0-670.gadc19233.el9.x86_ 24 MB/s | 17 MB 00:00 2026-02-18T06:19:25.695 INFO:teuthology.orchestra.run.trial031.stdout:(13/95): ceph-volume-20.2.0-670.gadc19233.el9.n 4.9 MB/s | 297 kB 00:00 2026-02-18T06:19:25.711 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/95 2026-02-18T06:19:25.714 INFO:teuthology.orchestra.run.trial031.stdout:(14/95): ceph-mgr-diskprediction-local-20.2.0-6 18 MB/s | 7.4 MB 00:00 2026-02-18T06:19:25.717 INFO:teuthology.orchestra.run.trial031.stdout:(15/95): cephadm-20.2.0-670.gadc19233.el9.noarc 42 MB/s | 1.0 MB 00:00 2026-02-18T06:19:25.721 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/95 2026-02-18T06:19:25.726 INFO:teuthology.orchestra.run.trial130.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 6/95 2026-02-18T06:19:25.729 INFO:teuthology.orchestra.run.trial130.stdout: Installing : lua-5.4.4-4.el9.x86_64 7/95 2026-02-18T06:19:25.733 INFO:teuthology.orchestra.run.trial130.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 8/95 2026-02-18T06:19:25.756 INFO:teuthology.orchestra.run.trial130.stdout: Installing : unzip-6.0-59.el9.x86_64 9/95 2026-02-18T06:19:25.769 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 10/95 2026-02-18T06:19:25.773 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 11/95 2026-02-18T06:19:25.777 INFO:teuthology.orchestra.run.trial031.stdout:(16/95): ledmon-libs-1.1.0-3.el9.x86_64.rpm 647 kB/s | 40 kB 00:00 2026-02-18T06:19:25.779 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 12/95 2026-02-18T06:19:25.780 INFO:teuthology.orchestra.run.trial031.stdout:(17/95): libconfig-1.7.2-9.el9.x86_64.rpm 1.1 MB/s | 72 kB 00:00 2026-02-18T06:19:25.780 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 13/95 2026-02-18T06:19:25.791 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 14/95 2026-02-18T06:19:25.797 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 15/95 2026-02-18T06:19:25.805 INFO:teuthology.orchestra.run.trial031.stdout:(18/95): cryptsetup-2.8.1-2.el9.x86_64.rpm 3.1 MB/s | 350 kB 00:00 2026-02-18T06:19:25.821 INFO:teuthology.orchestra.run.trial130.stdout: Installing : zip-3.0-35.el9.x86_64 16/95 2026-02-18T06:19:25.824 INFO:teuthology.orchestra.run.trial031.stdout:(19/95): libquadmath-11.5.0-14.el9.x86_64.rpm 4.1 MB/s | 184 kB 00:00 2026-02-18T06:19:25.825 INFO:teuthology.orchestra.run.trial130.stdout: Installing : luarocks-3.9.2-5.el9.noarch 17/95 2026-02-18T06:19:25.830 INFO:teuthology.orchestra.run.trial031.stdout:(20/95): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-18T06:19:25.832 INFO:teuthology.orchestra.run.trial130.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 18/95 2026-02-18T06:19:25.853 INFO:teuthology.orchestra.run.trial031.stdout:(21/95): python3-ply-3.11-14.el9.noarch.rpm 4.4 MB/s | 106 kB 00:00 2026-02-18T06:19:25.863 INFO:teuthology.orchestra.run.trial031.stdout:(22/95): libgfortran-11.5.0-14.el9.x86_64.rpm 9.0 MB/s | 794 kB 00:00 2026-02-18T06:19:25.877 INFO:teuthology.orchestra.run.trial031.stdout:(23/95): python3-pycparser-2.20-6.el9.noarch.rp 5.6 MB/s | 135 kB 00:00 2026-02-18T06:19:25.882 INFO:teuthology.orchestra.run.trial130.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 19/95 2026-02-18T06:19:25.887 INFO:teuthology.orchestra.run.trial031.stdout:(24/95): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-18T06:19:25.895 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 20/95 2026-02-18T06:19:25.897 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-rsa-4.9-2.el9.noarch 21/95 2026-02-18T06:19:25.900 INFO:teuthology.orchestra.run.trial031.stdout:(25/95): python3-requests-2.25.1-10.el9.noarch. 5.3 MB/s | 126 kB 00:00 2026-02-18T06:19:25.902 INFO:teuthology.orchestra.run.trial130.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 22/95 2026-02-18T06:19:25.907 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 23/95 2026-02-18T06:19:25.908 INFO:teuthology.orchestra.run.trial031.stdout:(26/95): python3-cryptography-36.0.1-5.el9.x86_ 15 MB/s | 1.2 MB 00:00 2026-02-18T06:19:25.911 INFO:teuthology.orchestra.run.trial031.stdout:(27/95): python3-urllib3-1.26.5-7.el9.noarch.rp 8.8 MB/s | 218 kB 00:00 2026-02-18T06:19:25.912 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 24/95 2026-02-18T06:19:25.915 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 25/95 2026-02-18T06:19:25.923 INFO:teuthology.orchestra.run.trial031.stdout:(28/95): unzip-6.0-59.el9.x86_64.rpm 7.7 MB/s | 182 kB 00:00 2026-02-18T06:19:25.929 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 26/95 2026-02-18T06:19:25.932 INFO:teuthology.orchestra.run.trial031.stdout:(29/95): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-18T06:19:25.934 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 27/95 2026-02-18T06:19:25.939 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 28/95 2026-02-18T06:19:25.950 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 29/95 2026-02-18T06:19:25.959 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 30/95 2026-02-18T06:19:25.965 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 31/95 2026-02-18T06:19:25.985 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 32/95 2026-02-18T06:19:25.996 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 33/95 2026-02-18T06:19:26.011 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 34/95 2026-02-18T06:19:26.016 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 35/95 2026-02-18T06:19:26.023 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 36/95 2026-02-18T06:19:26.063 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 37/95 2026-02-18T06:19:26.319 INFO:teuthology.orchestra.run.trial031.stdout:(30/95): flexiblas-3.0.4-9.el9.x86_64.rpm 73 kB/s | 30 kB 00:00 2026-02-18T06:19:26.335 INFO:teuthology.orchestra.run.trial031.stdout:(31/95): flexiblas-openblas-openmp-3.0.4-9.el9. 37 kB/s | 15 kB 00:00 2026-02-18T06:19:26.390 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 38/95 2026-02-18T06:19:26.400 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 39/95 2026-02-18T06:19:26.404 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 40/95 2026-02-18T06:19:26.410 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 41/95 2026-02-18T06:19:26.412 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 42/95 2026-02-18T06:19:26.440 INFO:teuthology.orchestra.run.trial130.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 43/95 2026-02-18T06:19:26.484 INFO:teuthology.orchestra.run.trial130.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 44/95 2026-02-18T06:19:26.490 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 45/95 2026-02-18T06:19:26.494 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 46/95 2026-02-18T06:19:26.499 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 47/95 2026-02-18T06:19:26.503 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 48/95 2026-02-18T06:19:26.510 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 49/95 2026-02-18T06:19:26.514 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 50/95 2026-02-18T06:19:26.541 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 51/95 2026-02-18T06:19:26.551 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 52/95 2026-02-18T06:19:26.558 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 53/95 2026-02-18T06:19:26.559 INFO:teuthology.orchestra.run.trial031.stdout:(32/95): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 1.0 MB/s | 246 kB 00:00 2026-02-18T06:19:26.565 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 54/95 2026-02-18T06:19:26.575 INFO:teuthology.orchestra.run.trial031.stdout:(33/95): lua-5.4.4-4.el9.x86_64.rpm 784 kB/s | 188 kB 00:00 2026-02-18T06:19:26.599 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 55/95 2026-02-18T06:19:26.642 INFO:teuthology.orchestra.run.trial031.stdout:(34/95): openblas-0.3.29-1.el9.x86_64.rpm 507 kB/s | 42 kB 00:00 2026-02-18T06:19:26.821 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 56/95 2026-02-18T06:19:26.845 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 57/95 2026-02-18T06:19:26.849 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 58/95 2026-02-18T06:19:26.882 INFO:teuthology.orchestra.run.trial031.stdout:(35/95): protobuf-3.14.0-17.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2026-02-18T06:19:26.907 INFO:teuthology.orchestra.run.trial130.stdout: Installing : openblas-0.3.29-1.el9.x86_64 59/95 2026-02-18T06:19:26.909 INFO:teuthology.orchestra.run.trial130.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 60/95 2026-02-18T06:19:26.930 INFO:teuthology.orchestra.run.trial130.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 61/95 2026-02-18T06:19:26.958 INFO:teuthology.orchestra.run.trial031.stdout:(36/95): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 2.9 MB/s | 3.0 MB 00:01 2026-02-18T06:19:27.042 INFO:teuthology.orchestra.run.trial031.stdout:(37/95): python3-devel-3.9.25-2.el9.x86_64.rpm 2.8 MB/s | 244 kB 00:00 2026-02-18T06:19:27.127 INFO:teuthology.orchestra.run.trial031.stdout:(38/95): python3-jinja2-2.11.3-8.el9.noarch.rpm 2.9 MB/s | 249 kB 00:00 2026-02-18T06:19:27.211 INFO:teuthology.orchestra.run.trial031.stdout:(39/95): python3-jmespath-1.0.1-1.el9.noarch.rp 567 kB/s | 48 kB 00:00 2026-02-18T06:19:27.253 INFO:teuthology.orchestra.run.trial130.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 62/95 2026-02-18T06:19:27.297 INFO:teuthology.orchestra.run.trial031.stdout:(40/95): python3-libstoragemgmt-1.10.1-1.el9.x8 2.0 MB/s | 177 kB 00:00 2026-02-18T06:19:27.326 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 63/95 2026-02-18T06:19:27.367 INFO:teuthology.orchestra.run.trial031.stdout:(41/95): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2026-02-18T06:19:27.455 INFO:teuthology.orchestra.run.trial031.stdout:(42/95): openblas-openmp-0.3.29-1.el9.x86_64.rp 6.0 MB/s | 5.3 MB 00:00 2026-02-18T06:19:27.455 INFO:teuthology.orchestra.run.trial031.stdout:(43/95): python3-markupsafe-1.1.1-12.el9.x86_64 396 kB/s | 35 kB 00:00 2026-02-18T06:19:27.463 INFO:teuthology.orchestra.run.trial031.stdout:(44/95): python3-lxml-4.6.5-3.el9.x86_64.rpm 7.4 MB/s | 1.2 MB 00:00 2026-02-18T06:19:27.545 INFO:teuthology.orchestra.run.trial031.stdout:(45/95): python3-numpy-f2py-1.23.5-2.el9.x86_64 4.8 MB/s | 442 kB 00:00 2026-02-18T06:19:27.547 INFO:teuthology.orchestra.run.trial031.stdout:(46/95): python3-packaging-20.9-5.el9.noarch.rp 913 kB/s | 77 kB 00:00 2026-02-18T06:19:27.632 INFO:teuthology.orchestra.run.trial031.stdout:(47/95): python3-numpy-1.23.5-2.el9.x86_64.rpm 35 MB/s | 6.1 MB 00:00 2026-02-18T06:19:27.633 INFO:teuthology.orchestra.run.trial031.stdout:(48/95): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 159 kB 00:00 2026-02-18T06:19:27.634 INFO:teuthology.orchestra.run.trial031.stdout:(49/95): python3-protobuf-3.14.0-17.el9.noarch. 2.9 MB/s | 267 kB 00:00 2026-02-18T06:19:27.717 INFO:teuthology.orchestra.run.trial031.stdout:(50/95): python3-pyasn1-modules-0.4.8-6.el9.noa 3.2 MB/s | 279 kB 00:00 2026-02-18T06:19:27.717 INFO:teuthology.orchestra.run.trial031.stdout:(51/95): python3-requests-oauthlib-1.3.0-12.el9 640 kB/s | 54 kB 00:00 2026-02-18T06:19:27.800 INFO:teuthology.orchestra.run.trial031.stdout:(52/95): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.2 MB/s | 189 kB 00:00 2026-02-18T06:19:27.800 INFO:teuthology.orchestra.run.trial031.stdout:(53/95): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2026-02-18T06:19:27.822 INFO:teuthology.orchestra.run.trial031.stdout:(54/95): lua-devel-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 22 kB 00:00 2026-02-18T06:19:27.883 INFO:teuthology.orchestra.run.trial031.stdout:(55/95): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-18T06:19:27.920 INFO:teuthology.orchestra.run.trial031.stdout:(56/95): protobuf-compiler-3.14.0-17.el9.x86_64 8.7 MB/s | 862 kB 00:00 2026-02-18T06:19:27.996 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 64/95 2026-02-18T06:19:28.004 INFO:teuthology.orchestra.run.trial031.stdout:(57/95): grpc-data-1.46.7-10.el9.noarch.rpm 239 kB/s | 19 kB 00:00 2026-02-18T06:19:28.013 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 65/95 2026-02-18T06:19:28.023 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 66/95 2026-02-18T06:19:28.030 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-packaging-20.9-5.el9.noarch 67/95 2026-02-18T06:19:28.043 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-ply-3.11-14.el9.noarch 68/95 2026-02-18T06:19:28.059 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 69/95 2026-02-18T06:19:28.065 INFO:teuthology.orchestra.run.trial031.stdout:(58/95): abseil-cpp-20211102.0-4.el9.x86_64.rpm 3.0 MB/s | 551 kB 00:00 2026-02-18T06:19:28.069 INFO:teuthology.orchestra.run.trial031.stdout:(59/95): luarocks-3.9.2-5.el9.noarch.rpm 2.3 MB/s | 151 kB 00:00 2026-02-18T06:19:28.089 INFO:teuthology.orchestra.run.trial031.stdout:(60/95): python3-autocommand-2.2.2-8.el9.noarch 1.4 MB/s | 29 kB 00:00 2026-02-18T06:19:28.096 INFO:teuthology.orchestra.run.trial031.stdout:(61/95): python3-asyncssh-2.13.2-5.el9.noarch.r 17 MB/s | 548 kB 00:00 2026-02-18T06:19:28.112 INFO:teuthology.orchestra.run.trial031.stdout:(62/95): python3-backports-tarfile-1.2.0-1.el9. 2.6 MB/s | 60 kB 00:00 2026-02-18T06:19:28.115 INFO:teuthology.orchestra.run.trial031.stdout:(63/95): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.2 MB/s | 43 kB 00:00 2026-02-18T06:19:28.132 INFO:teuthology.orchestra.run.trial031.stdout:(64/95): python3-cachetools-4.2.4-1.el9.noarch. 1.6 MB/s | 32 kB 00:00 2026-02-18T06:19:28.133 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 70/95 2026-02-18T06:19:28.134 INFO:teuthology.orchestra.run.trial031.stdout:(65/95): python3-certifi-2023.05.07-4.el9.noarc 750 kB/s | 14 kB 00:00 2026-02-18T06:19:28.142 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 71/95 2026-02-18T06:19:28.163 INFO:teuthology.orchestra.run.trial031.stdout:(66/95): python3-scipy-1.9.3-2.el9.x86_64.rpm 36 MB/s | 19 MB 00:00 2026-02-18T06:19:28.165 INFO:teuthology.orchestra.run.trial031.stdout:(67/95): python3-cherrypy-18.6.1-2.el9.noarch.r 12 MB/s | 358 kB 00:00 2026-02-18T06:19:28.165 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 72/95 2026-02-18T06:19:28.166 INFO:teuthology.orchestra.run.trial031.stdout:(68/95): python3-cheroot-10.0.1-4.el9.noarch.rp 5.1 MB/s | 173 kB 00:00 2026-02-18T06:19:28.193 INFO:teuthology.orchestra.run.trial031.stdout:(69/95): python3-grpcio-tools-1.46.7-10.el9.x86 5.1 MB/s | 144 kB 00:00 2026-02-18T06:19:28.195 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 73/95 2026-02-18T06:19:28.220 INFO:teuthology.orchestra.run.trial031.stdout:(70/95): python3-influxdb-5.3.1-1.el9.noarch.rp 5.2 MB/s | 139 kB 00:00 2026-02-18T06:19:28.224 INFO:teuthology.orchestra.run.trial031.stdout:(71/95): python3-grpcio-1.46.7-10.el9.x86_64.rp 35 MB/s | 2.0 MB 00:00 2026-02-18T06:19:28.241 INFO:teuthology.orchestra.run.trial031.stdout:(72/95): python3-isodate-0.6.1-3.el9.noarch.rpm 2.6 MB/s | 56 kB 00:00 2026-02-18T06:19:28.243 INFO:teuthology.orchestra.run.trial031.stdout:(73/95): python3-jaraco-8.2.1-3.el9.noarch.rpm 565 kB/s | 11 kB 00:00 2026-02-18T06:19:28.245 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 74/95 2026-02-18T06:19:28.251 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 75/95 2026-02-18T06:19:28.255 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 76/95 2026-02-18T06:19:28.256 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 77/95 2026-02-18T06:19:28.260 INFO:teuthology.orchestra.run.trial031.stdout:(74/95): python3-jaraco-classes-3.2.1-5.el9.noa 912 kB/s | 18 kB 00:00 2026-02-18T06:19:28.262 INFO:teuthology.orchestra.run.trial031.stdout:(75/95): python3-jaraco-collections-3.0.0-8.el9 1.2 MB/s | 23 kB 00:00 2026-02-18T06:19:28.273 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-18T06:19:28.273 INFO:teuthology.orchestra.run.trial130.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-18T06:19:28.273 INFO:teuthology.orchestra.run.trial130.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-18T06:19:28.273 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:28.280 INFO:teuthology.orchestra.run.trial031.stdout:(76/95): python3-jaraco-context-6.0.1-3.el9.noa 1.0 MB/s | 20 kB 00:00 2026-02-18T06:19:28.280 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-18T06:19:28.281 INFO:teuthology.orchestra.run.trial031.stdout:(77/95): python3-jaraco-functools-3.5.0-2.el9.n 1.0 MB/s | 19 kB 00:00 2026-02-18T06:19:28.299 INFO:teuthology.orchestra.run.trial031.stdout:(78/95): python3-jaraco-text-4.0.0-2.el9.noarch 1.3 MB/s | 26 kB 00:00 2026-02-18T06:19:28.303 INFO:teuthology.orchestra.run.trial031.stdout:(79/95): python3-google-auth-2.45.0-1.el9.noarc 1.8 MB/s | 254 kB 00:00 2026-02-18T06:19:28.306 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-18T06:19:28.306 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-18T06:19:28.306 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:28.307 INFO:teuthology.orchestra.run.trial031.stdout:(80/95): python3-kubernetes-26.1.0-3.el9.noarch 39 MB/s | 1.0 MB 00:00 2026-02-18T06:19:28.323 INFO:teuthology.orchestra.run.trial031.stdout:(81/95): python3-more-itertools-8.12.0-2.el9.no 3.3 MB/s | 79 kB 00:00 2026-02-18T06:19:28.326 INFO:teuthology.orchestra.run.trial031.stdout:(82/95): python3-natsort-7.1.1-5.el9.noarch.rpm 2.9 MB/s | 58 kB 00:00 2026-02-18T06:19:28.327 INFO:teuthology.orchestra.run.trial031.stdout:(83/95): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.7 MB/s | 86 kB 00:00 2026-02-18T06:19:28.340 INFO:teuthology.orchestra.run.trial130.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 79/95 2026-02-18T06:19:28.342 INFO:teuthology.orchestra.run.trial031.stdout:(84/95): python3-portend-3.1.0-2.el9.noarch.rpm 848 kB/s | 16 kB 00:00 2026-02-18T06:19:28.346 INFO:teuthology.orchestra.run.trial031.stdout:(85/95): python3-pyOpenSSL-21.0.0-1.el9.noarch. 4.5 MB/s | 90 kB 00:00 2026-02-18T06:19:28.346 INFO:teuthology.orchestra.run.trial031.stdout:(86/95): python3-repoze-lru-0.7-16.el9.noarch.r 1.6 MB/s | 31 kB 00:00 2026-02-18T06:19:28.365 INFO:teuthology.orchestra.run.trial031.stdout:(87/95): python3-rsa-4.9-2.el9.noarch.rpm 3.0 MB/s | 59 kB 00:00 2026-02-18T06:19:28.371 INFO:teuthology.orchestra.run.trial031.stdout:(88/95): python3-saml-1.16.0-1.el9.noarch.rpm 4.9 MB/s | 125 kB 00:00 2026-02-18T06:19:28.371 INFO:teuthology.orchestra.run.trial031.stdout:(89/95): python3-routes-2.5.1-5.el9.noarch.rpm 6.3 MB/s | 188 kB 00:00 2026-02-18T06:19:28.384 INFO:teuthology.orchestra.run.trial031.stdout:(90/95): python3-tempora-5.0.0-2.el9.noarch.rpm 1.8 MB/s | 36 kB 00:00 2026-02-18T06:19:28.394 INFO:teuthology.orchestra.run.trial031.stdout:(91/95): python3-typing-extensions-4.15.0-1.el9 3.7 MB/s | 86 kB 00:00 2026-02-18T06:19:28.394 INFO:teuthology.orchestra.run.trial031.stdout:(92/95): python3-websocket-client-1.2.3-2.el9.n 3.9 MB/s | 90 kB 00:00 2026-02-18T06:19:28.402 INFO:teuthology.orchestra.run.trial130.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 80/95 2026-02-18T06:19:28.403 INFO:teuthology.orchestra.run.trial031.stdout:(93/95): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.5 MB/s | 48 kB 00:00 2026-02-18T06:19:28.405 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 81/95 2026-02-18T06:19:28.413 INFO:teuthology.orchestra.run.trial031.stdout:(94/95): python3-xmltodict-0.12.0-15.el9.noarch 1.1 MB/s | 22 kB 00:00 2026-02-18T06:19:28.414 INFO:teuthology.orchestra.run.trial031.stdout:(95/95): python3-zc-lockfile-2.0-10.el9.noarch. 1.0 MB/s | 20 kB 00:00 2026-02-18T06:19:28.417 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:28.417 INFO:teuthology.orchestra.run.trial031.stdout:Total 25 MB/s | 101 MB 00:04 2026-02-18T06:19:28.418 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 81/95 2026-02-18T06:19:28.418 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:28.418 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-18T06:19:28.418 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:28.857 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:28.894 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:28.894 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:29.089 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 82/95 2026-02-18T06:19:29.115 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 82/95 2026-02-18T06:19:29.115 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:29.115 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-18T06:19:29.115 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-18T06:19:29.115 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-18T06:19:29.116 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:29.179 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: cephadm-2:20.2.0-670.gadc19233.el9.noarch 83/95 2026-02-18T06:19:29.181 INFO:teuthology.orchestra.run.trial130.stdout: Installing : cephadm-2:20.2.0-670.gadc19233.el9.noarch 83/95 2026-02-18T06:19:29.187 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-prometheus-alerts-2:20.2.0-670.gadc19233.el9. 84/95 2026-02-18T06:19:29.210 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-grafana-dashboards-2:20.2.0-670.gadc19233.el9 85/95 2026-02-18T06:19:29.212 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 86/95 2026-02-18T06:19:29.325 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:29.325 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:29.779 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:29.781 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/95 2026-02-18T06:19:29.785 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/95 2026-02-18T06:19:29.806 INFO:teuthology.orchestra.run.trial031.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/95 2026-02-18T06:19:29.839 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/95 2026-02-18T06:19:29.848 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/95 2026-02-18T06:19:29.854 INFO:teuthology.orchestra.run.trial031.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 6/95 2026-02-18T06:19:29.856 INFO:teuthology.orchestra.run.trial031.stdout: Installing : lua-5.4.4-4.el9.x86_64 7/95 2026-02-18T06:19:29.860 INFO:teuthology.orchestra.run.trial031.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 8/95 2026-02-18T06:19:29.884 INFO:teuthology.orchestra.run.trial031.stdout: Installing : unzip-6.0-59.el9.x86_64 9/95 2026-02-18T06:19:29.897 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 10/95 2026-02-18T06:19:29.901 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 11/95 2026-02-18T06:19:29.907 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 12/95 2026-02-18T06:19:29.908 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 13/95 2026-02-18T06:19:29.920 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 14/95 2026-02-18T06:19:29.925 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 15/95 2026-02-18T06:19:29.949 INFO:teuthology.orchestra.run.trial031.stdout: Installing : zip-3.0-35.el9.x86_64 16/95 2026-02-18T06:19:29.953 INFO:teuthology.orchestra.run.trial031.stdout: Installing : luarocks-3.9.2-5.el9.noarch 17/95 2026-02-18T06:19:29.959 INFO:teuthology.orchestra.run.trial031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 18/95 2026-02-18T06:19:30.010 INFO:teuthology.orchestra.run.trial031.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 19/95 2026-02-18T06:19:30.023 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 20/95 2026-02-18T06:19:30.025 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 21/95 2026-02-18T06:19:30.030 INFO:teuthology.orchestra.run.trial031.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 22/95 2026-02-18T06:19:30.035 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 23/95 2026-02-18T06:19:30.040 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 24/95 2026-02-18T06:19:30.043 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 25/95 2026-02-18T06:19:30.057 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 26/95 2026-02-18T06:19:30.061 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 27/95 2026-02-18T06:19:30.066 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 28/95 2026-02-18T06:19:30.078 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 29/95 2026-02-18T06:19:30.087 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 30/95 2026-02-18T06:19:30.093 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 31/95 2026-02-18T06:19:30.112 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 32/95 2026-02-18T06:19:30.123 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 33/95 2026-02-18T06:19:30.139 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 34/95 2026-02-18T06:19:30.143 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 35/95 2026-02-18T06:19:30.151 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 36/95 2026-02-18T06:19:30.191 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 37/95 2026-02-18T06:19:30.356 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 86/95 2026-02-18T06:19:30.362 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noar 87/95 2026-02-18T06:19:30.513 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 38/95 2026-02-18T06:19:30.523 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 39/95 2026-02-18T06:19:30.527 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 40/95 2026-02-18T06:19:30.533 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 41/95 2026-02-18T06:19:30.535 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 42/95 2026-02-18T06:19:30.563 INFO:teuthology.orchestra.run.trial031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 43/95 2026-02-18T06:19:30.606 INFO:teuthology.orchestra.run.trial031.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 44/95 2026-02-18T06:19:30.613 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 45/95 2026-02-18T06:19:30.616 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 46/95 2026-02-18T06:19:30.622 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 47/95 2026-02-18T06:19:30.625 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 48/95 2026-02-18T06:19:30.633 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 49/95 2026-02-18T06:19:30.636 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 50/95 2026-02-18T06:19:30.664 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 51/95 2026-02-18T06:19:30.673 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 52/95 2026-02-18T06:19:30.679 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 53/95 2026-02-18T06:19:30.686 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 54/95 2026-02-18T06:19:30.720 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 55/95 2026-02-18T06:19:30.818 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noar 87/95 2026-02-18T06:19:30.819 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mgr-diskprediction-local-2:20.2.0-670.gadc192 88/95 2026-02-18T06:19:30.833 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-670.gadc192 88/95 2026-02-18T06:19:30.833 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noar 89/95 2026-02-18T06:19:30.891 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noar 89/95 2026-02-18T06:19:30.934 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mgr-modules-core-2:20.2.0-670.gadc19233.el9.n 90/95 2026-02-18T06:19:30.935 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 91/95 2026-02-18T06:19:30.943 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 56/95 2026-02-18T06:19:30.957 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 91/95 2026-02-18T06:19:30.957 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:30.957 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-18T06:19:30.957 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-18T06:19:30.957 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-18T06:19:30.958 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:30.966 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 57/95 2026-02-18T06:19:30.969 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 92/95 2026-02-18T06:19:30.970 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 58/95 2026-02-18T06:19:31.028 INFO:teuthology.orchestra.run.trial031.stdout: Installing : openblas-0.3.29-1.el9.x86_64 59/95 2026-02-18T06:19:31.030 INFO:teuthology.orchestra.run.trial031.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 60/95 2026-02-18T06:19:31.051 INFO:teuthology.orchestra.run.trial031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 61/95 2026-02-18T06:19:31.193 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 92/95 2026-02-18T06:19:31.195 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 93/95 2026-02-18T06:19:31.217 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 93/95 2026-02-18T06:19:31.217 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:31.218 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-18T06:19:31.218 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-18T06:19:31.218 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-18T06:19:31.218 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:31.313 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 94/95 2026-02-18T06:19:31.334 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 94/95 2026-02-18T06:19:31.334 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:31.334 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-18T06:19:31.334 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-18T06:19:31.334 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-18T06:19:31.335 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:31.353 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-2:20.2.0-670.gadc19233.el9.x86_64 95/95 2026-02-18T06:19:31.374 INFO:teuthology.orchestra.run.trial031.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 62/95 2026-02-18T06:19:31.447 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 63/95 2026-02-18T06:19:32.060 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-2:20.2.0-670.gadc19233.el9.x86_64 95/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-2:20.2.0-670.gadc19233.el9.x86_64 1/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 2/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 3/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 4/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 5/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-670.gadc19233.el9 6/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 7/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noar 8/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-670.gadc192 9/95 2026-02-18T06:19:32.061 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noar 10/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-670.gadc19233.el9.n 11/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 12/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-670.gadc19233.el9. 13/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 14/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : cephadm-2:20.2.0-670.gadc19233.el9.noarch 15/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 16/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/95 2026-02-18T06:19:32.062 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : zip-3.0-35.el9.x86_64 29/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/95 2026-02-18T06:19:32.063 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 39/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 41/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 42/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 43/95 2026-02-18T06:19:32.064 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/95 2026-02-18T06:19:32.065 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/95 2026-02-18T06:19:32.066 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 81/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 82/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 83/95 2026-02-18T06:19:32.067 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 84/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 85/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 86/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 87/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 88/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 89/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 90/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 91/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 92/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 93/95 2026-02-18T06:19:32.068 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 94/95 2026-02-18T06:19:32.115 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 64/95 2026-02-18T06:19:32.124 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 95/95 2026-02-18T06:19:32.124 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:32.124 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:32.124 INFO:teuthology.orchestra.run.trial130.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-18T06:19:32.124 INFO:teuthology.orchestra.run.trial130.stdout: ceph-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-grafana-dashboards-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-diskprediction-local-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-modules-core-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:32.125 INFO:teuthology.orchestra.run.trial130.stdout: ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: ceph-prometheus-alerts-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: cephadm-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-18T06:19:32.126 INFO:teuthology.orchestra.run.trial130.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: lua-5.4.4-4.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-18T06:19:32.127 INFO:teuthology.orchestra.run.trial130.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-18T06:19:32.128 INFO:teuthology.orchestra.run.trial130.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-18T06:19:32.129 INFO:teuthology.orchestra.run.trial130.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-ply-3.11-14.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-18T06:19:32.130 INFO:teuthology.orchestra.run.trial130.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-18T06:19:32.131 INFO:teuthology.orchestra.run.trial130.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial130.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial130.stdout: unzip-6.0-59.el9.x86_64 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial130.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial130.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial130.stdout: zip-3.0-35.el9.x86_64 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:32.132 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 65/95 2026-02-18T06:19:32.142 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 66/95 2026-02-18T06:19:32.149 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 67/95 2026-02-18T06:19:32.162 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-ply-3.11-14.el9.noarch 68/95 2026-02-18T06:19:32.178 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 69/95 2026-02-18T06:19:32.179 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-base 2026-02-18T06:19:32.252 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 70/95 2026-02-18T06:19:32.261 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 71/95 2026-02-18T06:19:32.284 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 72/95 2026-02-18T06:19:32.313 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 73/95 2026-02-18T06:19:32.364 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 74/95 2026-02-18T06:19:32.370 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 75/95 2026-02-18T06:19:32.373 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 76/95 2026-02-18T06:19:32.374 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 77/95 2026-02-18T06:19:32.392 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-18T06:19:32.392 INFO:teuthology.orchestra.run.trial031.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-18T06:19:32.392 INFO:teuthology.orchestra.run.trial031.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-18T06:19:32.392 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:32.399 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-18T06:19:32.424 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 78/95 2026-02-18T06:19:32.425 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-18T06:19:32.425 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:32.459 INFO:teuthology.orchestra.run.trial031.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 79/95 2026-02-18T06:19:32.485 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:34 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:32.521 INFO:teuthology.orchestra.run.trial031.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 80/95 2026-02-18T06:19:32.524 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 81/95 2026-02-18T06:19:32.538 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 81/95 2026-02-18T06:19:32.538 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:32.538 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-18T06:19:32.539 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:32.549 INFO:teuthology.orchestra.run.trial130.stdout:Package ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:32.567 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:32.568 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:32.568 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:32.588 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install cephadm 2026-02-18T06:19:32.900 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:34 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:32.964 INFO:teuthology.orchestra.run.trial130.stdout:Package cephadm-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:32.982 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:32.983 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:32.983 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:33.004 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-immutable-object-cache 2026-02-18T06:19:33.229 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 82/95 2026-02-18T06:19:33.255 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 82/95 2026-02-18T06:19:33.255 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:33.255 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-18T06:19:33.255 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-18T06:19:33.255 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-18T06:19:33.255 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:33.313 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: cephadm-2:20.2.0-670.gadc19233.el9.noarch 83/95 2026-02-18T06:19:33.315 INFO:teuthology.orchestra.run.trial031.stdout: Installing : cephadm-2:20.2.0-670.gadc19233.el9.noarch 83/95 2026-02-18T06:19:33.316 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:35 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:33.322 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-prometheus-alerts-2:20.2.0-670.gadc19233.el9. 84/95 2026-02-18T06:19:33.345 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-grafana-dashboards-2:20.2.0-670.gadc19233.el9 85/95 2026-02-18T06:19:33.347 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 86/95 2026-02-18T06:19:33.399 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout: Package Arch Version Repo Size 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-670.gadc19233.el9 ceph 154 k 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:33.400 INFO:teuthology.orchestra.run.trial130.stdout:Install 1 Package 2026-02-18T06:19:33.401 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:33.401 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 154 k 2026-02-18T06:19:33.401 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 447 k 2026-02-18T06:19:33.401 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:33.485 INFO:teuthology.orchestra.run.trial130.stdout:ceph-immutable-object-cache-20.2.0-670.gadc1923 1.8 MB/s | 154 kB 00:00 2026-02-18T06:19:33.486 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:33.486 INFO:teuthology.orchestra.run.trial130.stdout:Total 1.7 MB/s | 154 kB 00:00 2026-02-18T06:19:33.486 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:33.490 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:33.491 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:33.523 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:33.523 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:33.578 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:33.580 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-immutable-object-cache-2:20.2.0-670.gadc19233.e 1/1 2026-02-18T06:19:33.601 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-670.gadc19233.e 1/1 2026-02-18T06:19:33.601 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:33.601 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-18T06:19:33.602 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:33.914 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-670.gadc19233.e 1/1 2026-02-18T06:19:33.914 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:33.914 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:33.915 INFO:teuthology.orchestra.run.trial130.stdout: ceph-immutable-object-cache-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:33.915 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:33.915 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:33.943 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-mgr 2026-02-18T06:19:34.249 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:36 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:34.313 INFO:teuthology.orchestra.run.trial130.stdout:Package ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:34.331 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:34.332 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:34.332 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:34.352 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-mgr-dashboard 2026-02-18T06:19:34.502 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 86/95 2026-02-18T06:19:34.507 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noar 87/95 2026-02-18T06:19:34.656 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:36 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:34.719 INFO:teuthology.orchestra.run.trial130.stdout:Package ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:34.737 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:34.738 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:34.738 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:34.758 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-18T06:19:34.966 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noar 87/95 2026-02-18T06:19:34.967 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mgr-diskprediction-local-2:20.2.0-670.gadc192 88/95 2026-02-18T06:19:34.980 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-670.gadc192 88/95 2026-02-18T06:19:34.981 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noar 89/95 2026-02-18T06:19:35.036 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noar 89/95 2026-02-18T06:19:35.066 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:37 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:35.080 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mgr-modules-core-2:20.2.0-670.gadc19233.el9.n 90/95 2026-02-18T06:19:35.081 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 91/95 2026-02-18T06:19:35.103 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 91/95 2026-02-18T06:19:35.103 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:35.103 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-18T06:19:35.103 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-18T06:19:35.104 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-18T06:19:35.104 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:35.115 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 92/95 2026-02-18T06:19:35.129 INFO:teuthology.orchestra.run.trial130.stdout:Package ceph-mgr-diskprediction-local-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:35.147 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:35.147 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:35.148 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:35.168 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-mgr-rook 2026-02-18T06:19:35.339 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 92/95 2026-02-18T06:19:35.341 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 93/95 2026-02-18T06:19:35.363 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 93/95 2026-02-18T06:19:35.363 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:35.363 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-18T06:19:35.364 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-18T06:19:35.364 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-18T06:19:35.364 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:35.463 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 94/95 2026-02-18T06:19:35.478 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:37 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:35.484 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 94/95 2026-02-18T06:19:35.484 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:35.484 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-18T06:19:35.485 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-18T06:19:35.485 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-18T06:19:35.485 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:35.503 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-2:20.2.0-670.gadc19233.el9.x86_64 95/95 2026-02-18T06:19:35.544 INFO:teuthology.orchestra.run.trial130.stdout:Package ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:35.561 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:35.562 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:35.562 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:35.583 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-mgr-cephadm 2026-02-18T06:19:35.895 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:37 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:35.959 INFO:teuthology.orchestra.run.trial130.stdout:Package ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:35.976 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:35.977 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:35.977 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:35.998 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-fuse 2026-02-18T06:19:36.165 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-2:20.2.0-670.gadc19233.el9.x86_64 95/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-2:20.2.0-670.gadc19233.el9.x86_64 1/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 2/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 3/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 4/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 5/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-670.gadc19233.el9 6/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 7/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noar 8/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-670.gadc192 9/95 2026-02-18T06:19:36.166 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noar 10/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-670.gadc19233.el9.n 11/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 12/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-670.gadc19233.el9. 13/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 14/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : cephadm-2:20.2.0-670.gadc19233.el9.noarch 15/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 16/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/95 2026-02-18T06:19:36.167 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : zip-3.0-35.el9.x86_64 29/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/95 2026-02-18T06:19:36.168 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 39/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 41/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 42/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 43/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/95 2026-02-18T06:19:36.169 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/95 2026-02-18T06:19:36.170 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/95 2026-02-18T06:19:36.171 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 81/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 82/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 83/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 84/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 85/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 86/95 2026-02-18T06:19:36.172 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 87/95 2026-02-18T06:19:36.173 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 88/95 2026-02-18T06:19:36.173 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 89/95 2026-02-18T06:19:36.173 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 90/95 2026-02-18T06:19:36.173 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 91/95 2026-02-18T06:19:36.173 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 92/95 2026-02-18T06:19:36.173 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 93/95 2026-02-18T06:19:36.173 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 94/95 2026-02-18T06:19:36.230 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 95/95 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: ceph-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: ceph-grafana-dashboards-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mds-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.231 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-diskprediction-local-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-k8sevents-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-modules-core-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: ceph-mon-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: ceph-osd-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: ceph-prometheus-alerts-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: ceph-volume-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: cephadm-2:20.2.0-670.gadc19233.el9.noarch 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-18T06:19:36.232 INFO:teuthology.orchestra.run.trial031.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: lua-5.4.4-4.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-18T06:19:36.233 INFO:teuthology.orchestra.run.trial031.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-18T06:19:36.234 INFO:teuthology.orchestra.run.trial031.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-18T06:19:36.235 INFO:teuthology.orchestra.run.trial031.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-ply-3.11-14.el9.noarch 2026-02-18T06:19:36.236 INFO:teuthology.orchestra.run.trial031.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-18T06:19:36.237 INFO:teuthology.orchestra.run.trial031.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: unzip-6.0-59.el9.x86_64 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-18T06:19:36.238 INFO:teuthology.orchestra.run.trial031.stdout: zip-3.0-35.el9.x86_64 2026-02-18T06:19:36.239 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:36.239 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:36.286 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-base 2026-02-18T06:19:36.307 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:38 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:36.390 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout: Package Architecture Version Repository Size 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout: ceph-fuse x86_64 2:20.2.0-670.gadc19233.el9 ceph 942 k 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:36.391 INFO:teuthology.orchestra.run.trial130.stdout:Install 1 Package 2026-02-18T06:19:36.392 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:36.392 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 942 k 2026-02-18T06:19:36.392 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 2.7 M 2026-02-18T06:19:36.392 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:36.510 INFO:teuthology.orchestra.run.trial130.stdout:ceph-fuse-20.2.0-670.gadc19233.el9.x86_64.rpm 7.7 MB/s | 942 kB 00:00 2026-02-18T06:19:36.511 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:36.511 INFO:teuthology.orchestra.run.trial130.stdout:Total 7.7 MB/s | 942 kB 00:00 2026-02-18T06:19:36.511 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:36.515 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:36.515 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:36.549 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:36.549 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:36.599 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:37 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:36.639 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:36.656 INFO:teuthology.orchestra.run.trial130.stdout: Installing : ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:36.663 INFO:teuthology.orchestra.run.trial031.stdout:Package ceph-base-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:36.681 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:36.681 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:36.681 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:36.703 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install cephadm 2026-02-18T06:19:36.884 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:36.941 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:36.941 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:36.941 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:36.941 INFO:teuthology.orchestra.run.trial130.stdout: ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:36.941 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:36.941 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:36.967 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install ceph-volume 2026-02-18T06:19:37.012 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:38 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:37.077 INFO:teuthology.orchestra.run.trial031.stdout:Package cephadm-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:37.095 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:37.095 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:37.095 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:37.117 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-immutable-object-cache 2026-02-18T06:19:37.281 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:39 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:37.344 INFO:teuthology.orchestra.run.trial130.stdout:Package ceph-volume-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:37.362 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:37.362 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:37.362 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:37.384 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install librados-devel 2026-02-18T06:19:37.424 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:38 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:37.506 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:37.506 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout: Package Arch Version Repo Size 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-670.gadc19233.el9 ceph 154 k 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout:Install 1 Package 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:37.507 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 154 k 2026-02-18T06:19:37.508 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 447 k 2026-02-18T06:19:37.508 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:37.592 INFO:teuthology.orchestra.run.trial031.stdout:ceph-immutable-object-cache-20.2.0-670.gadc1923 1.8 MB/s | 154 kB 00:00 2026-02-18T06:19:37.593 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:37.594 INFO:teuthology.orchestra.run.trial031.stdout:Total 1.7 MB/s | 154 kB 00:00 2026-02-18T06:19:37.594 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:37.598 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:37.598 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:37.631 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:37.631 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:37.686 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:37.688 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-immutable-object-cache-2:20.2.0-670.gadc19233.e 1/1 2026-02-18T06:19:37.693 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:39 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:37.710 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-670.gadc19233.e 1/1 2026-02-18T06:19:37.710 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:37.710 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-18T06:19:37.710 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:37.774 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:37.775 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:37.775 INFO:teuthology.orchestra.run.trial130.stdout: Package Arch Version Repo Size 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout: librados-devel x86_64 2:20.2.0-670.gadc19233.el9 ceph 126 k 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:Install 1 Package 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 126 k 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 449 k 2026-02-18T06:19:37.776 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:37.860 INFO:teuthology.orchestra.run.trial130.stdout:librados-devel-20.2.0-670.gadc19233.el9.x86_64. 1.4 MB/s | 126 kB 00:00 2026-02-18T06:19:37.861 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:37.861 INFO:teuthology.orchestra.run.trial130.stdout:Total 1.4 MB/s | 126 kB 00:00 2026-02-18T06:19:37.861 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:37.864 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:37.864 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:37.895 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:37.895 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:37.949 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:37.965 INFO:teuthology.orchestra.run.trial130.stdout: Installing : librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:38.027 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-670.gadc19233.e 1/1 2026-02-18T06:19:38.027 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:38.027 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:38.027 INFO:teuthology.orchestra.run.trial031.stdout: ceph-immutable-object-cache-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:38.027 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:38.028 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:38.043 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:38.054 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-mgr 2026-02-18T06:19:38.100 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:38.101 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:38.101 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:38.101 INFO:teuthology.orchestra.run.trial130.stdout: librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:38.101 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:38.101 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:38.129 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install libcephfs2 2026-02-18T06:19:38.371 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:39 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:38.437 INFO:teuthology.orchestra.run.trial031.stdout:Package ceph-mgr-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:38.439 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:38.454 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:38.455 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:38.455 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:38.477 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-mgr-dashboard 2026-02-18T06:19:38.504 INFO:teuthology.orchestra.run.trial130.stdout:Package libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:38.522 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:38.522 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:38.523 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:38.544 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install libcephfs-devel 2026-02-18T06:19:38.784 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:39 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:38.848 INFO:teuthology.orchestra.run.trial031.stdout:Package ceph-mgr-dashboard-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:38.855 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:38.866 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:38.867 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:38.867 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:38.888 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-18T06:19:38.937 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:38.937 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout: Package Arch Version Repo Size 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs-devel x86_64 2:20.2.0-670.gadc19233.el9 ceph 34 k 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout:Install 1 Package 2026-02-18T06:19:38.938 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:38.939 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 34 k 2026-02-18T06:19:38.939 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 155 k 2026-02-18T06:19:38.939 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:39.001 INFO:teuthology.orchestra.run.trial130.stdout:libcephfs-devel-20.2.0-670.gadc19233.el9.x86_64 547 kB/s | 34 kB 00:00 2026-02-18T06:19:39.001 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:39.001 INFO:teuthology.orchestra.run.trial130.stdout:Total 540 kB/s | 34 kB 00:00 2026-02-18T06:19:39.001 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:39.003 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:39.003 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:39.005 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:39.005 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:39.027 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:39.043 INFO:teuthology.orchestra.run.trial130.stdout: Installing : libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:39.101 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:39.157 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:39.157 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:39.157 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:39.157 INFO:teuthology.orchestra.run.trial130.stdout: libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:39.157 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:39.157 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:39.186 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install librados2 2026-02-18T06:19:39.195 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:39.259 INFO:teuthology.orchestra.run.trial031.stdout:Package ceph-mgr-diskprediction-local-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:39.277 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:39.278 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:39.278 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:39.300 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-mgr-rook 2026-02-18T06:19:39.497 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:41 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:39.560 INFO:teuthology.orchestra.run.trial130.stdout:Package librados2-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:39.578 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:39.579 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:39.579 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:39.601 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install librbd1 2026-02-18T06:19:39.606 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:39.670 INFO:teuthology.orchestra.run.trial031.stdout:Package ceph-mgr-rook-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:39.687 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:39.688 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:39.688 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:39.709 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-mgr-cephadm 2026-02-18T06:19:39.912 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:41 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:39.977 INFO:teuthology.orchestra.run.trial130.stdout:Package librbd1-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:39.995 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:39.995 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:39.995 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:40.017 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install python3-rados 2026-02-18T06:19:40.024 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:41 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:40.087 INFO:teuthology.orchestra.run.trial031.stdout:Package ceph-mgr-cephadm-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:40.105 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:40.106 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:40.106 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:40.127 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-fuse 2026-02-18T06:19:40.322 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:42 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:40.386 INFO:teuthology.orchestra.run.trial130.stdout:Package python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:40.404 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:40.405 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:40.405 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:40.426 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install python3-rgw 2026-02-18T06:19:40.433 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:41 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:40.515 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:40.516 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:40.516 INFO:teuthology.orchestra.run.trial031.stdout: Package Architecture Version Repository Size 2026-02-18T06:19:40.516 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:40.516 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:40.516 INFO:teuthology.orchestra.run.trial031.stdout: ceph-fuse x86_64 2:20.2.0-670.gadc19233.el9 ceph 942 k 2026-02-18T06:19:40.516 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:40.517 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:40.517 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:40.517 INFO:teuthology.orchestra.run.trial031.stdout:Install 1 Package 2026-02-18T06:19:40.517 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:40.517 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 942 k 2026-02-18T06:19:40.517 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 2.7 M 2026-02-18T06:19:40.517 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:40.635 INFO:teuthology.orchestra.run.trial031.stdout:ceph-fuse-20.2.0-670.gadc19233.el9.x86_64.rpm 7.8 MB/s | 942 kB 00:00 2026-02-18T06:19:40.636 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:40.636 INFO:teuthology.orchestra.run.trial031.stdout:Total 7.7 MB/s | 942 kB 00:00 2026-02-18T06:19:40.636 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:40.641 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:40.641 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:40.674 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:40.674 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:40.733 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:42 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:40.763 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:40.780 INFO:teuthology.orchestra.run.trial031.stdout: Installing : ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:40.797 INFO:teuthology.orchestra.run.trial130.stdout:Package python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:40.815 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:40.816 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:40.816 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:40.837 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install python3-cephfs 2026-02-18T06:19:41.012 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:41.068 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:41.068 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:41.068 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:41.069 INFO:teuthology.orchestra.run.trial031.stdout: ceph-fuse-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:41.069 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:41.069 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:41.096 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install ceph-volume 2026-02-18T06:19:41.149 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:43 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:41.212 INFO:teuthology.orchestra.run.trial130.stdout:Package python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:41.230 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:41.231 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:41.231 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:41.253 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install python3-rbd 2026-02-18T06:19:41.405 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:42 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:41.472 INFO:teuthology.orchestra.run.trial031.stdout:Package ceph-volume-2:20.2.0-670.gadc19233.el9.noarch is already installed. 2026-02-18T06:19:41.490 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:41.491 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:41.491 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:41.513 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install librados-devel 2026-02-18T06:19:41.557 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:43 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:41.622 INFO:teuthology.orchestra.run.trial130.stdout:Package python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:41.639 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:41.640 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:41.640 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:41.661 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install rbd-fuse 2026-02-18T06:19:41.826 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:42 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:41.907 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout: Package Arch Version Repo Size 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout: librados-devel x86_64 2:20.2.0-670.gadc19233.el9 ceph 126 k 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout:Install 1 Package 2026-02-18T06:19:41.908 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:41.909 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 126 k 2026-02-18T06:19:41.909 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 449 k 2026-02-18T06:19:41.909 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:41.972 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:43 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:41.993 INFO:teuthology.orchestra.run.trial031.stdout:librados-devel-20.2.0-670.gadc19233.el9.x86_64. 1.5 MB/s | 126 kB 00:00 2026-02-18T06:19:41.993 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:41.994 INFO:teuthology.orchestra.run.trial031.stdout:Total 1.4 MB/s | 126 kB 00:00 2026-02-18T06:19:41.994 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:41.997 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:41.997 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:42.028 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:42.028 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:42.055 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:42.056 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:42.056 INFO:teuthology.orchestra.run.trial130.stdout: Package Architecture Version Repository Size 2026-02-18T06:19:42.056 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:42.056 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:42.056 INFO:teuthology.orchestra.run.trial130.stdout: rbd-fuse x86_64 2:20.2.0-670.gadc19233.el9 ceph 91 k 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout:Install 1 Package 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 91 k 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 238 k 2026-02-18T06:19:42.057 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:42.083 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:42.099 INFO:teuthology.orchestra.run.trial031.stdout: Installing : librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:42.129 INFO:teuthology.orchestra.run.trial130.stdout:rbd-fuse-20.2.0-670.gadc19233.el9.x86_64.rpm 1.2 MB/s | 91 kB 00:00 2026-02-18T06:19:42.130 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:42.130 INFO:teuthology.orchestra.run.trial130.stdout:Total 1.2 MB/s | 91 kB 00:00 2026-02-18T06:19:42.130 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:42.135 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:42.135 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:42.165 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:42.166 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:42.186 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:42.220 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:42.237 INFO:teuthology.orchestra.run.trial130.stdout: Installing : rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:42.244 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:42.244 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:42.244 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:42.244 INFO:teuthology.orchestra.run.trial031.stdout: librados-devel-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:42.244 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:42.244 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:42.273 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install libcephfs2 2026-02-18T06:19:42.318 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:42.375 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:42.375 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:42.375 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:42.375 INFO:teuthology.orchestra.run.trial130.stdout: rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:42.375 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:42.375 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:42.404 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install rbd-mirror 2026-02-18T06:19:42.592 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:43 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:42.657 INFO:teuthology.orchestra.run.trial031.stdout:Package libcephfs2-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:42.674 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:42.675 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:42.675 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:42.696 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install libcephfs-devel 2026-02-18T06:19:42.722 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:44 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:42.804 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:42.804 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout: Package Arch Version Repo Size 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout: rbd-mirror x86_64 2:20.2.0-670.gadc19233.el9 ceph 2.9 M 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout:Install 1 Package 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:42.805 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 2.9 M 2026-02-18T06:19:42.806 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 11 M 2026-02-18T06:19:42.806 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:42.949 INFO:teuthology.orchestra.run.trial130.stdout:rbd-mirror-20.2.0-670.gadc19233.el9.x86_64.rpm 20 MB/s | 2.9 MB 00:00 2026-02-18T06:19:42.950 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:42.950 INFO:teuthology.orchestra.run.trial130.stdout:Total 20 MB/s | 2.9 MB 00:00 2026-02-18T06:19:42.950 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:42.954 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:42.954 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:42.993 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:42.993 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:43.008 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:44 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:43.089 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:43.090 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:43.090 INFO:teuthology.orchestra.run.trial031.stdout: Package Arch Version Repo Size 2026-02-18T06:19:43.090 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:43.090 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:43.090 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs-devel x86_64 2:20.2.0-670.gadc19233.el9 ceph 34 k 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout:Install 1 Package 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 34 k 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 155 k 2026-02-18T06:19:43.091 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:43.153 INFO:teuthology.orchestra.run.trial031.stdout:libcephfs-devel-20.2.0-670.gadc19233.el9.x86_64 551 kB/s | 34 kB 00:00 2026-02-18T06:19:43.153 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:43.153 INFO:teuthology.orchestra.run.trial031.stdout:Total 543 kB/s | 34 kB 00:00 2026-02-18T06:19:43.153 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:43.156 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:43.156 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:43.158 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:43.158 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:43.175 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:43.177 INFO:teuthology.orchestra.run.trial130.stdout: Installing : rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:43.179 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:43.196 INFO:teuthology.orchestra.run.trial031.stdout: Installing : libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:43.199 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:43.199 INFO:teuthology.orchestra.run.trial130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:43.199 INFO:teuthology.orchestra.run.trial130.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-18T06:19:43.199 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-18T06:19:43.199 INFO:teuthology.orchestra.run.trial130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-18T06:19:43.199 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:43.252 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:43.308 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:43.308 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:43.308 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:43.308 INFO:teuthology.orchestra.run.trial031.stdout: libcephfs-devel-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:43.308 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:43.308 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:43.334 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install librados2 2026-02-18T06:19:43.504 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:43.504 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:43.504 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:43.505 INFO:teuthology.orchestra.run.trial130.stdout: rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:43.505 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:43.505 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:43.532 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install rbd-nbd 2026-02-18T06:19:43.645 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:44 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:43.709 INFO:teuthology.orchestra.run.trial031.stdout:Package librados2-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:43.726 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:43.727 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:43.727 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:43.748 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install librbd1 2026-02-18T06:19:43.847 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:45 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:43.929 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:43.929 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout: Package Architecture Version Repository Size 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout:Installing: 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout: rbd-nbd x86_64 2:20.2.0-670.gadc19233.el9 ceph 180 k 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout:Transaction Summary 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout:================================================================================ 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout:Install 1 Package 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:43.930 INFO:teuthology.orchestra.run.trial130.stdout:Total download size: 180 k 2026-02-18T06:19:43.931 INFO:teuthology.orchestra.run.trial130.stdout:Installed size: 498 k 2026-02-18T06:19:43.931 INFO:teuthology.orchestra.run.trial130.stdout:Downloading Packages: 2026-02-18T06:19:44.014 INFO:teuthology.orchestra.run.trial130.stdout:rbd-nbd-20.2.0-670.gadc19233.el9.x86_64.rpm 2.1 MB/s | 180 kB 00:00 2026-02-18T06:19:44.015 INFO:teuthology.orchestra.run.trial130.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:44.015 INFO:teuthology.orchestra.run.trial130.stdout:Total 2.1 MB/s | 180 kB 00:00 2026-02-18T06:19:44.015 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction check 2026-02-18T06:19:44.020 INFO:teuthology.orchestra.run.trial130.stdout:Transaction check succeeded. 2026-02-18T06:19:44.020 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction test 2026-02-18T06:19:44.050 INFO:teuthology.orchestra.run.trial130.stdout:Transaction test succeeded. 2026-02-18T06:19:44.051 INFO:teuthology.orchestra.run.trial130.stdout:Running transaction 2026-02-18T06:19:44.060 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:45 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:44.106 INFO:teuthology.orchestra.run.trial130.stdout: Preparing : 1/1 2026-02-18T06:19:44.123 INFO:teuthology.orchestra.run.trial130.stdout: Installing : rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:44.124 INFO:teuthology.orchestra.run.trial031.stdout:Package librbd1-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:44.141 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:44.142 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:44.142 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:44.163 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install python3-rados 2026-02-18T06:19:44.204 INFO:teuthology.orchestra.run.trial130.stdout: Running scriptlet: rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:44.263 INFO:teuthology.orchestra.run.trial130.stdout: Verifying : rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:44.263 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:44.263 INFO:teuthology.orchestra.run.trial130.stdout:Installed: 2026-02-18T06:19:44.263 INFO:teuthology.orchestra.run.trial130.stdout: rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:44.263 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:44.263 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:44.292 DEBUG:teuthology.orchestra.run.trial130:> sudo yum -y install rbd-mirror 2026-02-18T06:19:44.472 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:45 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:44.535 INFO:teuthology.orchestra.run.trial031.stdout:Package python3-rados-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:44.553 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:44.553 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:44.554 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:44.574 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install python3-rgw 2026-02-18T06:19:44.603 INFO:teuthology.orchestra.run.trial130.stdout:Last metadata expiration check: 0:00:46 ago on Wed 18 Feb 2026 06:18:58 AM UTC. 2026-02-18T06:19:44.668 INFO:teuthology.orchestra.run.trial130.stdout:Package rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:44.685 INFO:teuthology.orchestra.run.trial130.stdout:Dependencies resolved. 2026-02-18T06:19:44.686 INFO:teuthology.orchestra.run.trial130.stdout:Nothing to do. 2026-02-18T06:19:44.686 INFO:teuthology.orchestra.run.trial130.stdout:Complete! 2026-02-18T06:19:44.711 DEBUG:teuthology.parallel:result is None 2026-02-18T06:19:44.889 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:45 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:44.952 INFO:teuthology.orchestra.run.trial031.stdout:Package python3-rgw-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:44.970 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:44.970 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:44.971 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:44.992 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install python3-cephfs 2026-02-18T06:19:45.303 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:46 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:45.366 INFO:teuthology.orchestra.run.trial031.stdout:Package python3-cephfs-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:45.384 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:45.385 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:45.385 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:45.406 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install python3-rbd 2026-02-18T06:19:45.713 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:46 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:45.776 INFO:teuthology.orchestra.run.trial031.stdout:Package python3-rbd-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:45.794 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:45.795 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:45.795 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:45.816 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install rbd-fuse 2026-02-18T06:19:46.130 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:47 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:46.212 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:46.213 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:46.213 INFO:teuthology.orchestra.run.trial031.stdout: Package Architecture Version Repository Size 2026-02-18T06:19:46.213 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:46.213 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:46.213 INFO:teuthology.orchestra.run.trial031.stdout: rbd-fuse x86_64 2:20.2.0-670.gadc19233.el9 ceph 91 k 2026-02-18T06:19:46.213 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:46.213 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:46.214 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:46.214 INFO:teuthology.orchestra.run.trial031.stdout:Install 1 Package 2026-02-18T06:19:46.214 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:46.214 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 91 k 2026-02-18T06:19:46.214 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 238 k 2026-02-18T06:19:46.214 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:46.285 INFO:teuthology.orchestra.run.trial031.stdout:rbd-fuse-20.2.0-670.gadc19233.el9.x86_64.rpm 1.2 MB/s | 91 kB 00:00 2026-02-18T06:19:46.286 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:46.286 INFO:teuthology.orchestra.run.trial031.stdout:Total 1.2 MB/s | 91 kB 00:00 2026-02-18T06:19:46.286 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:46.290 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:46.290 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:46.321 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:46.321 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:46.372 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:46.389 INFO:teuthology.orchestra.run.trial031.stdout: Installing : rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:46.477 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:46.533 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:46.534 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:46.534 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:46.534 INFO:teuthology.orchestra.run.trial031.stdout: rbd-fuse-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:46.534 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:46.534 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:46.562 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install rbd-mirror 2026-02-18T06:19:46.876 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:47 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:46.959 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:46.959 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout: Package Arch Version Repo Size 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout: rbd-mirror x86_64 2:20.2.0-670.gadc19233.el9 ceph 2.9 M 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout:Install 1 Package 2026-02-18T06:19:46.960 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:46.961 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 2.9 M 2026-02-18T06:19:46.961 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 11 M 2026-02-18T06:19:46.961 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:47.106 INFO:teuthology.orchestra.run.trial031.stdout:rbd-mirror-20.2.0-670.gadc19233.el9.x86_64.rpm 20 MB/s | 2.9 MB 00:00 2026-02-18T06:19:47.107 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:47.107 INFO:teuthology.orchestra.run.trial031.stdout:Total 20 MB/s | 2.9 MB 00:00 2026-02-18T06:19:47.107 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:47.112 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:47.112 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:47.151 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:47.151 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:47.331 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:47.332 INFO:teuthology.orchestra.run.trial031.stdout: Installing : rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:47.355 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:47.355 INFO:teuthology.orchestra.run.trial031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-18T06:19:47.355 INFO:teuthology.orchestra.run.trial031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-18T06:19:47.355 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-18T06:19:47.355 INFO:teuthology.orchestra.run.trial031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-18T06:19:47.355 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:47.669 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:47.669 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:47.670 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:47.670 INFO:teuthology.orchestra.run.trial031.stdout: rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:47.670 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:47.670 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:47.698 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install rbd-nbd 2026-02-18T06:19:48.010 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:49 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:48.092 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout: Package Architecture Version Repository Size 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout:Installing: 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout: rbd-nbd x86_64 2:20.2.0-670.gadc19233.el9 ceph 180 k 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout:Transaction Summary 2026-02-18T06:19:48.093 INFO:teuthology.orchestra.run.trial031.stdout:================================================================================ 2026-02-18T06:19:48.094 INFO:teuthology.orchestra.run.trial031.stdout:Install 1 Package 2026-02-18T06:19:48.094 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:48.094 INFO:teuthology.orchestra.run.trial031.stdout:Total download size: 180 k 2026-02-18T06:19:48.094 INFO:teuthology.orchestra.run.trial031.stdout:Installed size: 498 k 2026-02-18T06:19:48.094 INFO:teuthology.orchestra.run.trial031.stdout:Downloading Packages: 2026-02-18T06:19:48.177 INFO:teuthology.orchestra.run.trial031.stdout:rbd-nbd-20.2.0-670.gadc19233.el9.x86_64.rpm 2.1 MB/s | 180 kB 00:00 2026-02-18T06:19:48.178 INFO:teuthology.orchestra.run.trial031.stdout:-------------------------------------------------------------------------------- 2026-02-18T06:19:48.178 INFO:teuthology.orchestra.run.trial031.stdout:Total 2.0 MB/s | 180 kB 00:00 2026-02-18T06:19:48.178 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction check 2026-02-18T06:19:48.183 INFO:teuthology.orchestra.run.trial031.stdout:Transaction check succeeded. 2026-02-18T06:19:48.183 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction test 2026-02-18T06:19:48.214 INFO:teuthology.orchestra.run.trial031.stdout:Transaction test succeeded. 2026-02-18T06:19:48.214 INFO:teuthology.orchestra.run.trial031.stdout:Running transaction 2026-02-18T06:19:48.270 INFO:teuthology.orchestra.run.trial031.stdout: Preparing : 1/1 2026-02-18T06:19:48.287 INFO:teuthology.orchestra.run.trial031.stdout: Installing : rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:48.365 INFO:teuthology.orchestra.run.trial031.stdout: Running scriptlet: rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:48.423 INFO:teuthology.orchestra.run.trial031.stdout: Verifying : rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 1/1 2026-02-18T06:19:48.423 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:48.423 INFO:teuthology.orchestra.run.trial031.stdout:Installed: 2026-02-18T06:19:48.423 INFO:teuthology.orchestra.run.trial031.stdout: rbd-nbd-2:20.2.0-670.gadc19233.el9.x86_64 2026-02-18T06:19:48.423 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:48.423 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:48.451 DEBUG:teuthology.orchestra.run.trial031:> sudo yum -y install rbd-mirror 2026-02-18T06:19:48.762 INFO:teuthology.orchestra.run.trial031.stdout:Last metadata expiration check: 0:00:49 ago on Wed 18 Feb 2026 06:18:59 AM UTC. 2026-02-18T06:19:48.826 INFO:teuthology.orchestra.run.trial031.stdout:Package rbd-mirror-2:20.2.0-670.gadc19233.el9.x86_64 is already installed. 2026-02-18T06:19:48.844 INFO:teuthology.orchestra.run.trial031.stdout:Dependencies resolved. 2026-02-18T06:19:48.845 INFO:teuthology.orchestra.run.trial031.stdout:Nothing to do. 2026-02-18T06:19:48.845 INFO:teuthology.orchestra.run.trial031.stdout:Complete! 2026-02-18T06:19:48.870 DEBUG:teuthology.parallel:result is None 2026-02-18T06:19:48.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T06:19:49.033 DEBUG:teuthology.orchestra.run.trial031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-18T06:19:49.054 INFO:teuthology.orchestra.run.trial031.stdout:20.2.0-670.gadc19233.el9 2026-02-18T06:19:49.055 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-670.gadc19233.el9 2026-02-18T06:19:49.055 INFO:teuthology.task.install:The correct ceph version 20.2.0-670.gadc19233 is installed. 2026-02-18T06:19:49.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T06:19:49.205 DEBUG:teuthology.orchestra.run.trial130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-18T06:19:49.226 INFO:teuthology.orchestra.run.trial130.stdout:20.2.0-670.gadc19233.el9 2026-02-18T06:19:49.226 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-670.gadc19233.el9 2026-02-18T06:19:49.227 INFO:teuthology.task.install:The correct ceph version 20.2.0-670.gadc19233 is installed. 2026-02-18T06:19:49.229 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-18T06:19:49.230 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:49.230 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-18T06:19:49.256 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:49.256 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-18T06:19:49.292 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-18T06:19:49.292 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:49.292 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/usr/bin/daemon-helper 2026-02-18T06:19:49.320 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-18T06:19:49.385 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:49.385 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/usr/bin/daemon-helper 2026-02-18T06:19:49.411 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-18T06:19:49.475 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-18T06:19:49.476 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:49.476 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-18T06:19:49.502 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-18T06:19:49.567 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:49.567 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-18T06:19:49.592 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-18T06:19:49.656 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-18T06:19:49.656 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:49.656 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/usr/bin/stdin-killer 2026-02-18T06:19:49.682 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-18T06:19:49.746 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:49.746 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/usr/bin/stdin-killer 2026-02-18T06:19:49.771 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-18T06:19:49.836 INFO:teuthology.run_tasks:Running task ceph... 2026-02-18T06:19:49.951 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-18T06:19:49.952 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod 777 /var/log/ceph 2026-02-18T06:19:49.954 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod 777 /var/log/ceph 2026-02-18T06:19:49.979 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-18T06:19:49.979 DEBUG:teuthology.orchestra.run.trial031:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-18T06:19:50.017 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-18T06:19:50.041 INFO:tasks.ceph:Creating extra log directories... 2026-02-18T06:19:50.041 DEBUG:teuthology.orchestra.run.trial031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-18T06:19:50.081 DEBUG:teuthology.orchestra.run.trial130:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-18T06:19:50.120 INFO:tasks.ceph:Creating ceph cluster cluster1... 2026-02-18T06:19:50.120 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-18T06:19:50.120 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885', 'branch': '20.2.1_2.17.26-baseline_2', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-policy-none}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '54885', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror.sh workunit to test the rbd-mirror daemon in journal mode'}], 'name': 'yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': '20.2.1_2.17.26-baseline_2'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': '20.2.1_2.17.26-baseline_2', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7878, 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': '20.2.1_2.17.26-baseline_2', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed', 'targets': {'trial031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHRNW7A2DvvWlKHRZNI6CLZ97SvCZ/eAtdF852Rm9tFeUL7CxpN5UeNDcxfpYyTzxAc0h0SLbexV5u04UgbTlx4=', 'trial130.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEwrfL57bUfT56dxS9dPm4meFgSKQcTf5bVd/MCniJ+6npZE+yMJCdieVstdiz0p0rICGb+5W5V0e2I6sZx+XsM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-17_21:26:40', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682'} 2026-02-18T06:19:50.121 DEBUG:teuthology.orchestra.run.trial031:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-18T06:19:50.145 DEBUG:teuthology.orchestra.run.trial130:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-18T06:19:50.161 DEBUG:teuthology.orchestra.run.trial031:> sudo install -d -m0777 -- /var/run/ceph 2026-02-18T06:19:50.201 DEBUG:teuthology.orchestra.run.trial130:> sudo install -d -m0777 -- /var/run/ceph 2026-02-18T06:19:50.227 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:50.227 DEBUG:teuthology.orchestra.run.trial031:> dd if=/scratch_devs of=/dev/stdout 2026-02-18T06:19:50.281 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-18T06:19:50.281 DEBUG:teuthology.orchestra.run.trial031:> stat /dev/vg_nvme/lv_1 2026-02-18T06:19:50.337 INFO:teuthology.orchestra.run.trial031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-18T06:19:50.337 INFO:teuthology.orchestra.run.trial031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:19:50.338 INFO:teuthology.orchestra.run.trial031.stdout:Device: 6h/6d Inode: 1290 Links: 1 2026-02-18T06:19:50.338 INFO:teuthology.orchestra.run.trial031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:19:50.338 INFO:teuthology.orchestra.run.trial031.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:19:50.338 INFO:teuthology.orchestra.run.trial031.stdout:Access: 2026-02-18 06:19:47.498138930 +0000 2026-02-18T06:19:50.338 INFO:teuthology.orchestra.run.trial031.stdout:Modify: 2026-02-18 06:19:05.244172011 +0000 2026-02-18T06:19:50.338 INFO:teuthology.orchestra.run.trial031.stdout:Change: 2026-02-18 06:19:05.244172011 +0000 2026-02-18T06:19:50.338 INFO:teuthology.orchestra.run.trial031.stdout: Birth: 2026-02-18 06:19:05.244172011 +0000 2026-02-18T06:19:50.339 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-18T06:19:50.400 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records in 2026-02-18T06:19:50.401 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records out 2026-02-18T06:19:50.401 INFO:teuthology.orchestra.run.trial031.stderr:512 bytes copied, 0.000227973 s, 2.2 MB/s 2026-02-18T06:19:50.402 DEBUG:teuthology.orchestra.run.trial031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-18T06:19:50.461 DEBUG:teuthology.orchestra.run.trial031:> stat /dev/vg_nvme/lv_2 2026-02-18T06:19:50.519 INFO:teuthology.orchestra.run.trial031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-18T06:19:50.519 INFO:teuthology.orchestra.run.trial031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:19:50.519 INFO:teuthology.orchestra.run.trial031.stdout:Device: 6h/6d Inode: 1311 Links: 1 2026-02-18T06:19:50.520 INFO:teuthology.orchestra.run.trial031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:19:50.520 INFO:teuthology.orchestra.run.trial031.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:19:50.520 INFO:teuthology.orchestra.run.trial031.stdout:Access: 2026-02-18 06:19:47.499138905 +0000 2026-02-18T06:19:50.520 INFO:teuthology.orchestra.run.trial031.stdout:Modify: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.520 INFO:teuthology.orchestra.run.trial031.stdout:Change: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.520 INFO:teuthology.orchestra.run.trial031.stdout: Birth: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.520 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-18T06:19:50.581 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records in 2026-02-18T06:19:50.582 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records out 2026-02-18T06:19:50.582 INFO:teuthology.orchestra.run.trial031.stderr:512 bytes copied, 0.000211272 s, 2.4 MB/s 2026-02-18T06:19:50.583 DEBUG:teuthology.orchestra.run.trial031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-18T06:19:50.642 DEBUG:teuthology.orchestra.run.trial031:> stat /dev/vg_nvme/lv_3 2026-02-18T06:19:50.698 INFO:teuthology.orchestra.run.trial031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout:Device: 6h/6d Inode: 1306 Links: 1 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout:Access: 2026-02-18 06:19:47.499138905 +0000 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout:Modify: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout:Change: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.699 INFO:teuthology.orchestra.run.trial031.stdout: Birth: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.700 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-18T06:19:50.761 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records in 2026-02-18T06:19:50.762 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records out 2026-02-18T06:19:50.762 INFO:teuthology.orchestra.run.trial031.stderr:512 bytes copied, 0.000231519 s, 2.2 MB/s 2026-02-18T06:19:50.763 DEBUG:teuthology.orchestra.run.trial031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-18T06:19:50.821 DEBUG:teuthology.orchestra.run.trial031:> stat /dev/vg_nvme/lv_4 2026-02-18T06:19:50.878 INFO:teuthology.orchestra.run.trial031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-18T06:19:50.878 INFO:teuthology.orchestra.run.trial031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:19:50.878 INFO:teuthology.orchestra.run.trial031.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-18T06:19:50.878 INFO:teuthology.orchestra.run.trial031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:19:50.878 INFO:teuthology.orchestra.run.trial031.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:19:50.878 INFO:teuthology.orchestra.run.trial031.stdout:Access: 2026-02-18 06:19:47.499138905 +0000 2026-02-18T06:19:50.879 INFO:teuthology.orchestra.run.trial031.stdout:Modify: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.879 INFO:teuthology.orchestra.run.trial031.stdout:Change: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.879 INFO:teuthology.orchestra.run.trial031.stdout: Birth: 2026-02-18 06:19:05.245171987 +0000 2026-02-18T06:19:50.879 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-18T06:19:50.940 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records in 2026-02-18T06:19:50.941 INFO:teuthology.orchestra.run.trial031.stderr:1+0 records out 2026-02-18T06:19:50.941 INFO:teuthology.orchestra.run.trial031.stderr:512 bytes copied, 0.000219507 s, 2.3 MB/s 2026-02-18T06:19:50.942 DEBUG:teuthology.orchestra.run.trial031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-18T06:19:50.999 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:19:50.999 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-18T06:19:50.999 INFO:tasks.ceph:Generating config... 2026-02-18T06:19:51.001 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client] log to stderr = False 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.002 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.003 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.003 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-18T06:19:51.003 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-18T06:19:51.003 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-18T06:19:51.004 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-18T06:19:51.004 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-18T06:19:51.004 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-18T06:19:51.004 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-18T06:19:51.004 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-18T06:19:51.004 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-18T06:19:51.004 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-18T06:19:51.004 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2026-02-18T06:19:51.004 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-18T06:19:51.004 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-18T06:19:51.004 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-18T06:19:51.005 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-18T06:19:51.005 INFO:tasks.ceph:Setting up cluster1.mon.a... 2026-02-18T06:19:51.006 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2026-02-18T06:19:51.072 INFO:teuthology.orchestra.run.trial031.stdout:creating /etc/ceph/cluster1.keyring 2026-02-18T06:19:51.075 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2026-02-18T06:19:51.152 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-18T06:19:51.216 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '10.20.193.31')] 2026-02-18T06:19:51.217 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.31', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-18T06:19:51.218 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:51.219 DEBUG:teuthology.orchestra.run.trial031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-18T06:19:51.274 DEBUG:teuthology.orchestra.run.trial031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.31 --print /home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:51.348 INFO:teuthology.orchestra.run.trial031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:51.348 INFO:teuthology.orchestra.run.trial031.stdout:monmaptool: generated fsid c34c7f09-e418-4318-892f-b221bab3e02e 2026-02-18T06:19:51.348 INFO:teuthology.orchestra.run.trial031.stdout:setting min_mon_release = tentacle 2026-02-18T06:19:51.348 INFO:teuthology.orchestra.run.trial031.stdout:epoch 0 2026-02-18T06:19:51.348 INFO:teuthology.orchestra.run.trial031.stdout:fsid c34c7f09-e418-4318-892f-b221bab3e02e 2026-02-18T06:19:51.348 INFO:teuthology.orchestra.run.trial031.stdout:last_changed 2026-02-18T06:19:51.346541+0000 2026-02-18T06:19:51.348 INFO:teuthology.orchestra.run.trial031.stdout:created 2026-02-18T06:19:51.346541+0000 2026-02-18T06:19:51.349 INFO:teuthology.orchestra.run.trial031.stdout:min_mon_release 20 (tentacle) 2026-02-18T06:19:51.349 INFO:teuthology.orchestra.run.trial031.stdout:election_strategy: 1 2026-02-18T06:19:51.349 INFO:teuthology.orchestra.run.trial031.stdout:0: [v2:10.20.193.31:3300/0,v1:10.20.193.31:6789/0] mon.a 2026-02-18T06:19:51.349 INFO:teuthology.orchestra.run.trial031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2026-02-18T06:19:51.350 DEBUG:teuthology.orchestra.run.trial031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-18T06:19:51.406 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID c34c7f09-e418-4318-892f-b221bab3e02e... 2026-02-18T06:19:51.407 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-18T06:19:51.448 DEBUG:teuthology.orchestra.run.trial130:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-18T06:19:51.487 INFO:teuthology.orchestra.run.trial031.stdout:[global] 2026-02-18T06:19:51.487 INFO:teuthology.orchestra.run.trial031.stdout: chdir = "" 2026-02-18T06:19:51.487 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-18T06:19:51.487 INFO:teuthology.orchestra.run.trial031.stdout: auth supported = cephx 2026-02-18T06:19:51.487 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.487 INFO:teuthology.orchestra.run.trial031.stdout: filestore xattr use omap = true 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: mon clock drift allowed = 1.000 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: osd crush chooseleaf type = 0 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: auth debug = true 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: ms die on old message = true 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: ms die on bug = true 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: mon pg warn max object skew = 0 2026-02-18T06:19:51.488 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: # disable pg_autoscaler by default for new pools 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: osd pool default size = 2 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: mon osd allow primary affinity = true 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: mon osd allow pg remap = true 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on legacy crush tunables = false 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on crush straw calc version zero = false 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on no sortbitwise = false 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on osd down out interval zero = false 2026-02-18T06:19:51.489 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on too few osds = false 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: mon_allow_pool_size_one = true 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: osd default data pool replay window = 5 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: mon allow pool delete = true 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: mon cluster log file level = debug 2026-02-18T06:19:51.490 INFO:teuthology.orchestra.run.trial031.stdout: debug asserts on shutdown = true 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: mon health detail to clog = false 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: mon host = 10.20.193.31 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: mon client directed command retry = 5 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on pool no app = False 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: ms inject socket failures = 5000 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: fsid = c34c7f09-e418-4318-892f-b221bab3e02e 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout:[osd] 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: osd journal size = 100 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: osd scrub load threshold = 5.0 2026-02-18T06:19:51.491 INFO:teuthology.orchestra.run.trial031.stdout: osd scrub max interval = 600 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd mclock profile = high_recovery_ops 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd mclock skip benchmark = true 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd recover clone overlap = true 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd recovery max chunk = 1048576 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd debug shutdown = true 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd debug op order = true 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd debug verify stray on activate = true 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd debug trim objects = true 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.492 INFO:teuthology.orchestra.run.trial031.stdout: osd open classes on start = true 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: osd debug pg log writeout = true 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: osd deep scrub update digest min age = 30 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: osd map max advance = 10 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: journal zero on create = true 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: filestore ondisk finisher threads = 3 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: filestore apply finisher threads = 3 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: bdev debug aio = true 2026-02-18T06:19:51.493 INFO:teuthology.orchestra.run.trial031.stdout: osd debug misdirected ops = true 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: bdev async discard = True 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: bdev enable discard = True 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: bluestore allocator = bitmap 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: bluestore block size = 96636764160 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: bluestore fsck on mount = True 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: debug bluefs = 1/20 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: debug bluestore = 1/20 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: debug ms = 1 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: debug osd = 20 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: debug rocksdb = 4/10 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: mon osd backfillfull_ratio = 0.85 2026-02-18T06:19:51.494 INFO:teuthology.orchestra.run.trial031.stdout: mon osd full ratio = 0.9 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: mon osd nearfull ratio = 0.8 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: osd failsafe full ratio = 0.95 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: osd objectstore = bluestore 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout:[mgr] 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: debug ms = 1 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: debug mgr = 20 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: debug mon = 20 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: debug auth = 20 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: mgr/telemetry/nag = false 2026-02-18T06:19:51.495 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout:[mon] 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: debug ms = 1 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: debug mon = 20 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: debug paxos = 20 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: debug auth = 20 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: mon data avail warn = 5 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: mon mgr mkfs grace = 240 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: mon osd reporter subtree level = osd 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: mon osd prime pg temp = true 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.496 INFO:teuthology.orchestra.run.trial031.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: auth mon ticket ttl = 660 # 11m 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: auth service ticket ttl = 240 # 4m 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: # don't complain about insecure global_id in the test suite 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: # 1m isn't quite enough 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: mon_down_mkfs_grace = 2m 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_filestore_osds = false 2026-02-18T06:19:51.497 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout:[client] 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: rgw cache enabled = true 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: rgw enable ops log = true 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: rgw enable usage log = true 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: debug rbd = 20 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: debug rbd_mirror = 15 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: log to stderr = False 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: rbd default features = 125 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout: rbd mirror image policy type = none 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout:[mon.a] 2026-02-18T06:19:51.498 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.0] 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.1] 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.2] 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.3] 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.4] 2026-02-18T06:19:51.499 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.500 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.500 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.5] 2026-02-18T06:19:51.500 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.500 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.500 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.6] 2026-02-18T06:19:51.500 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.500 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.502 INFO:teuthology.orchestra.run.trial130.stdout:[global] 2026-02-18T06:19:51.502 INFO:teuthology.orchestra.run.trial130.stdout: chdir = "" 2026-02-18T06:19:51.502 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-18T06:19:51.502 INFO:teuthology.orchestra.run.trial130.stdout: auth supported = cephx 2026-02-18T06:19:51.502 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.502 INFO:teuthology.orchestra.run.trial130.stdout: filestore xattr use omap = true 2026-02-18T06:19:51.502 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: mon clock drift allowed = 1.000 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: osd crush chooseleaf type = 0 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: auth debug = true 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: ms die on old message = true 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: ms die on bug = true 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: mon pg warn max object skew = 0 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.503 INFO:teuthology.orchestra.run.trial130.stdout: # disable pg_autoscaler by default for new pools 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: osd pool default size = 2 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon osd allow primary affinity = true 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon osd allow pg remap = true 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on legacy crush tunables = false 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on crush straw calc version zero = false 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on no sortbitwise = false 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on osd down out interval zero = false 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on too few osds = false 2026-02-18T06:19:51.504 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: mon_allow_pool_size_one = true 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: osd default data pool replay window = 5 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: mon allow pool delete = true 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: mon cluster log file level = debug 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: debug asserts on shutdown = true 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: mon health detail to clog = false 2026-02-18T06:19:51.505 INFO:teuthology.orchestra.run.trial130.stdout: mon host = 10.20.193.31 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: mon client directed command retry = 5 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on pool no app = False 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: ms inject socket failures = 5000 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: fsid = c34c7f09-e418-4318-892f-b221bab3e02e 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout:[osd] 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: osd journal size = 100 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: osd scrub load threshold = 5.0 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: osd scrub max interval = 600 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: osd mclock profile = high_recovery_ops 2026-02-18T06:19:51.506 INFO:teuthology.orchestra.run.trial130.stdout: osd mclock skip benchmark = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd recover clone overlap = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd recovery max chunk = 1048576 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd debug shutdown = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd debug op order = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd debug verify stray on activate = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd debug trim objects = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd open classes on start = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd debug pg log writeout = true 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.507 INFO:teuthology.orchestra.run.trial130.stdout: osd deep scrub update digest min age = 30 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: osd map max advance = 10 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: journal zero on create = true 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: filestore ondisk finisher threads = 3 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: filestore apply finisher threads = 3 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: bdev debug aio = true 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: osd debug misdirected ops = true 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: bdev async discard = True 2026-02-18T06:19:51.508 INFO:teuthology.orchestra.run.trial130.stdout: bdev enable discard = True 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: bluestore allocator = bitmap 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: bluestore block size = 96636764160 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: bluestore fsck on mount = True 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: debug bluefs = 1/20 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: debug bluestore = 1/20 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: debug ms = 1 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: debug osd = 20 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: debug rocksdb = 4/10 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: mon osd backfillfull_ratio = 0.85 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: mon osd full ratio = 0.9 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: mon osd nearfull ratio = 0.8 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: osd failsafe full ratio = 0.95 2026-02-18T06:19:51.509 INFO:teuthology.orchestra.run.trial130.stdout: osd objectstore = bluestore 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout:[mgr] 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: debug ms = 1 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: debug mgr = 20 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: debug mon = 20 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: debug auth = 20 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: mgr/telemetry/nag = false 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout:[mon] 2026-02-18T06:19:51.510 INFO:teuthology.orchestra.run.trial130.stdout: debug ms = 1 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: debug mon = 20 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: debug paxos = 20 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: debug auth = 20 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: mon data avail warn = 5 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: mon mgr mkfs grace = 240 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: mon osd reporter subtree level = osd 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: mon osd prime pg temp = true 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: auth mon ticket ttl = 660 # 11m 2026-02-18T06:19:51.511 INFO:teuthology.orchestra.run.trial130.stdout: auth service ticket ttl = 240 # 4m 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: # don't complain about insecure global_id in the test suite 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: # 1m isn't quite enough 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: mon_down_mkfs_grace = 2m 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_filestore_osds = false 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout:[client] 2026-02-18T06:19:51.512 INFO:teuthology.orchestra.run.trial130.stdout: rgw cache enabled = true 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: rgw enable ops log = true 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: rgw enable usage log = true 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: debug rbd = 20 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: debug rbd_mirror = 15 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: log to stderr = False 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: rbd default features = 125 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: rbd mirror image policy type = none 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout:[mon.a] 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.0] 2026-02-18T06:19:51.513 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.1] 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.2] 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.3] 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.4] 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.514 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.515 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.5] 2026-02-18T06:19:51.515 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.515 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.515 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.6] 2026-02-18T06:19:51.515 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:19:51.515 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:19:51.515 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2026-02-18T06:19:51.516 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2026-02-18T06:19:51.585 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-18T06:19:51.585 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:51.586 DEBUG:teuthology.orchestra.run.trial031:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2026-02-18T06:19:51.641 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:51.641 DEBUG:teuthology.orchestra.run.trial031:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2026-02-18T06:19:51.697 INFO:tasks.ceph:Sending monmap to node ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:19:51.698 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:51.698 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-18T06:19:51.698 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-18T06:19:51.772 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:51.772 DEBUG:teuthology.orchestra.run.trial031:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:51.828 INFO:tasks.ceph:Sending monmap to node ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:19:51.828 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:51.828 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-18T06:19:51.828 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-18T06:19:51.863 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:51.863 DEBUG:teuthology.orchestra.run.trial130:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:51.921 INFO:tasks.ceph:Setting up mon nodes... 2026-02-18T06:19:51.921 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-18T06:19:51.921 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-18T06:19:51.966 INFO:teuthology.orchestra.run.trial031.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-18T06:19:51.969 INFO:tasks.ceph:Setting up mds nodes... 2026-02-18T06:19:51.969 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-18T06:19:51.969 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2026-02-18T06:19:52.004 INFO:teuthology.orchestra.run.trial031.stdout:creating /etc/ceph/cluster1.client.0.keyring 2026-02-18T06:19:52.015 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2026-02-18T06:19:52.051 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2026-02-18T06:19:52.062 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-18T06:19:52.137 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-18T06:19:52.148 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-18T06:19:52.222 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-18T06:19:52.232 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-18T06:19:52.306 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-18T06:19:52.316 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-18T06:19:52.390 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-18T06:19:52.401 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-18T06:19:52.473 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-18T06:19:52.485 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-18T06:19:52.558 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-18T06:19:52.568 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-18T06:19:52.642 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-18T06:19:52.653 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-18T06:19:52.654 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-18T06:19:52.654 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2026-02-18T06:19:52.680 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:19:52.680 INFO:tasks.ceph:role: cluster1.osd.0 2026-02-18T06:19:52.680 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:19:52.680 DEBUG:teuthology.orchestra.run.trial031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-18T06:19:52.745 INFO:teuthology.orchestra.run.trial031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-18T06:19:52.745 INFO:teuthology.orchestra.run.trial031.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-18T06:19:52.745 INFO:teuthology.orchestra.run.trial031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-18T06:19:52.745 INFO:teuthology.orchestra.run.trial031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-18T06:19:52.745 INFO:teuthology.orchestra.run.trial031.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-18T06:19:52.745 INFO:teuthology.orchestra.run.trial031.stdout: = sunit=0 swidth=0 blks 2026-02-18T06:19:52.746 INFO:teuthology.orchestra.run.trial031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-18T06:19:52.746 INFO:teuthology.orchestra.run.trial031.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-18T06:19:52.746 INFO:teuthology.orchestra.run.trial031.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-18T06:19:52.746 INFO:teuthology.orchestra.run.trial031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-18T06:19:53.318 INFO:teuthology.orchestra.run.trial031.stdout:Discarding blocks...Done. 2026-02-18T06:19:53.321 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial031.front.sepia.ceph.com -o noatime 2026-02-18T06:19:53.321 DEBUG:teuthology.orchestra.run.trial031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2026-02-18T06:19:53.390 DEBUG:teuthology.orchestra.run.trial031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2026-02-18T06:19:53.458 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2026-02-18T06:19:53.522 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:19:53.522 INFO:tasks.ceph:role: cluster1.osd.1 2026-02-18T06:19:53.523 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:19:53.523 DEBUG:teuthology.orchestra.run.trial031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-18T06:19:53.587 INFO:teuthology.orchestra.run.trial031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-18T06:19:53.587 INFO:teuthology.orchestra.run.trial031.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout: = sunit=0 swidth=0 blks 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-18T06:19:53.588 INFO:teuthology.orchestra.run.trial031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-18T06:19:54.145 INFO:teuthology.orchestra.run.trial031.stdout:Discarding blocks...Done. 2026-02-18T06:19:54.147 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial031.front.sepia.ceph.com -o noatime 2026-02-18T06:19:54.147 DEBUG:teuthology.orchestra.run.trial031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2026-02-18T06:19:54.215 DEBUG:teuthology.orchestra.run.trial031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2026-02-18T06:19:54.283 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2026-02-18T06:19:54.349 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:19:54.350 INFO:tasks.ceph:role: cluster1.osd.2 2026-02-18T06:19:54.350 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:19:54.350 DEBUG:teuthology.orchestra.run.trial031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-18T06:19:54.415 INFO:teuthology.orchestra.run.trial031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-18T06:19:54.415 INFO:teuthology.orchestra.run.trial031.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-18T06:19:54.415 INFO:teuthology.orchestra.run.trial031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-18T06:19:54.416 INFO:teuthology.orchestra.run.trial031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-18T06:19:54.416 INFO:teuthology.orchestra.run.trial031.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-18T06:19:54.416 INFO:teuthology.orchestra.run.trial031.stdout: = sunit=0 swidth=0 blks 2026-02-18T06:19:54.416 INFO:teuthology.orchestra.run.trial031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-18T06:19:54.416 INFO:teuthology.orchestra.run.trial031.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-18T06:19:54.416 INFO:teuthology.orchestra.run.trial031.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-18T06:19:54.416 INFO:teuthology.orchestra.run.trial031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-18T06:19:54.975 INFO:teuthology.orchestra.run.trial031.stdout:Discarding blocks...Done. 2026-02-18T06:19:54.977 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial031.front.sepia.ceph.com -o noatime 2026-02-18T06:19:54.977 DEBUG:teuthology.orchestra.run.trial031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2026-02-18T06:19:55.046 DEBUG:teuthology.orchestra.run.trial031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2026-02-18T06:19:55.113 DEBUG:teuthology.orchestra.run.trial031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:55.193 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.190+0000 7f510f297900 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2026-02-18T06:19:55.194 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.190+0000 7f510f297900 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2026-02-18T06:19:55.194 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.190+0000 7f510f297900 -1 bdev(0x55ea2edf1800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2026-02-18T06:19:55.194 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.190+0000 7f510f297900 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2026-02-18T06:19:55.645 DEBUG:teuthology.orchestra.run.trial031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2026-02-18T06:19:55.709 DEBUG:teuthology.orchestra.run.trial031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:55.789 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.786+0000 7f4484ef8900 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2026-02-18T06:19:55.789 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.786+0000 7f4484ef8900 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2026-02-18T06:19:55.790 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.786+0000 7f4484ef8900 -1 bdev(0x55d87f749800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2026-02-18T06:19:55.790 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:55.786+0000 7f4484ef8900 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2026-02-18T06:19:56.228 DEBUG:teuthology.orchestra.run.trial031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2026-02-18T06:19:56.293 DEBUG:teuthology.orchestra.run.trial031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:56.370 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:56.367+0000 7f7a63c0f900 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2026-02-18T06:19:56.371 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:56.367+0000 7f7a63c0f900 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2026-02-18T06:19:56.371 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:56.367+0000 7f7a63c0f900 -1 bdev(0x5637559cb800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2026-02-18T06:19:56.371 INFO:teuthology.orchestra.run.trial031.stderr:2026-02-18T06:19:56.367+0000 7f7a63c0f900 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2026-02-18T06:19:56.846 DEBUG:teuthology.orchestra.run.trial031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2026-02-18T06:19:56.910 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-18T06:19:56.911 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:56.911 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2026-02-18T06:19:56.976 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:56.976 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2026-02-18T06:19:57.039 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:57.040 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2026-02-18T06:19:57.104 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:57.105 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2026-02-18T06:19:57.168 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:57.168 DEBUG:teuthology.orchestra.run.trial031:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2026-02-18T06:19:57.224 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.224 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2026-02-18T06:19:57.243 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.243 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2026-02-18T06:19:57.299 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.299 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2026-02-18T06:19:57.355 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.355 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2026-02-18T06:19:57.414 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.414 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2026-02-18T06:19:57.470 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.470 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2026-02-18T06:19:57.527 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.527 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2026-02-18T06:19:57.586 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:19:57.586 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2026-02-18T06:19:57.642 INFO:tasks.ceph:Adding keys to all mons... 2026-02-18T06:19:57.643 DEBUG:teuthology.orchestra.run.trial031:> sudo tee -a /etc/ceph/cluster1.keyring 2026-02-18T06:19:57.666 INFO:teuthology.orchestra.run.trial031.stdout:[mgr.x] 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAHWpVpv8qNORAAvL9rD4DR8tsYa+PZsrMSAg== 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout:[osd.0] 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout: key = AQALWpVpoa9zCxAAauGJU9igC6A5h+aCZQ3Zjw== 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout:[osd.1] 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout: key = AQALWpVp0gv1LhAAzfevB/H7Fvca/h+EaQuFTw== 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout:[osd.2] 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAMWpVpEhQCFhAAB8tgnDLPpU3ItmrVgFed3w== 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout:[client.0] 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpgukvABAAfbIyOrxMRMV70wqUxOd46A== 2026-02-18T06:19:57.667 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror] 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVp1n0VAxAAmfwRY6GefheReh7l824hLQ== 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.0] 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpUeAyCBAAK7O9/L3Caf/+xGXNz1wlFQ== 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.1] 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpFKZHDRAAMbnGo7HW52CSq5VLpIVi7Q== 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.2] 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpisZDEhAAmYxw6HD0NrD/RkCKJOON2A== 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.3] 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpvTxOFxAAb06so6zFGY8xZMII2wqzOg== 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.4] 2026-02-18T06:19:57.668 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpeSxDHBAAuQsO/vgyPZB9w71SRGuePQ== 2026-02-18T06:19:57.669 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.5] 2026-02-18T06:19:57.669 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpDkJGIRAAkjMFj/1dIV7rNKFFtPg3gg== 2026-02-18T06:19:57.669 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.6] 2026-02-18T06:19:57.669 INFO:teuthology.orchestra.run.trial031.stdout: key = AQAIWpVpf3JPJhAAzUnvPuAVHlMxl7KuSltkIA== 2026-02-18T06:19:57.670 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-18T06:19:57.751 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-18T06:19:57.830 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-18T06:19:57.910 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-18T06:19:57.991 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.073 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.155 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.236 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.320 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.401 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.482 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.564 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.646 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:19:58.727 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-18T06:19:58.727 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2026-02-18T06:19:58.792 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2026-02-18T06:19:58.874 DEBUG:teuthology.orchestra.run.trial031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2026-02-18T06:19:58.940 DEBUG:teuthology.orchestra.run.trial031:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2026-02-18T06:19:58.996 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2026-02-18T06:19:58.996 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2026-02-18T06:19:58.997 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2026-02-18T06:19:59.038 INFO:tasks.ceph.cluster1.mon.a:Started 2026-02-18T06:19:59.038 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2026-02-18T06:19:59.039 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2026-02-18T06:19:59.039 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2026-02-18T06:19:59.041 INFO:tasks.ceph.cluster1.mgr.x:Started 2026-02-18T06:19:59.042 DEBUG:tasks.ceph:set 0 configs 2026-02-18T06:19:59.042 DEBUG:teuthology.orchestra.run.trial031:> sudo ceph --cluster cluster1 config dump 2026-02-18T06:19:59.144 INFO:teuthology.orchestra.run.trial031.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-18T06:19:59.152 INFO:tasks.ceph:Setting crush tunables to default 2026-02-18T06:19:59.153 DEBUG:teuthology.orchestra.run.trial031:> sudo ceph --cluster cluster1 osd crush tunables default 2026-02-18T06:19:59.255 INFO:teuthology.orchestra.run.trial031.stderr:adjusted tunables profile to default 2026-02-18T06:19:59.262 INFO:tasks.ceph:check_enable_crimson: False 2026-02-18T06:19:59.262 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2026-02-18T06:19:59.263 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:59.263 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2026-02-18T06:19:59.287 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:59.288 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2026-02-18T06:19:59.350 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:19:59.351 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2026-02-18T06:19:59.416 DEBUG:teuthology.orchestra.run.trial031:> sudo ceph --cluster cluster1 osd new 269a8c5d-3e3d-44c0-969c-686555b67a10 0 2026-02-18T06:19:59.561 INFO:teuthology.orchestra.run.trial031.stdout:0 2026-02-18T06:19:59.570 DEBUG:teuthology.orchestra.run.trial031:> sudo ceph --cluster cluster1 osd new 91fe9ddc-6f8a-4a7f-ae09-18bea6a1597b 1 2026-02-18T06:19:59.676 INFO:teuthology.orchestra.run.trial031.stdout:1 2026-02-18T06:19:59.685 DEBUG:teuthology.orchestra.run.trial031:> sudo ceph --cluster cluster1 osd new d87b9373-e7c3-4a6e-b065-c5ee208d646f 2 2026-02-18T06:19:59.791 INFO:teuthology.orchestra.run.trial031.stdout:2 2026-02-18T06:19:59.799 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2026-02-18T06:19:59.799 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2026-02-18T06:19:59.801 INFO:tasks.ceph.cluster1.osd.0:Started 2026-02-18T06:19:59.802 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2026-02-18T06:19:59.802 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2026-02-18T06:19:59.804 INFO:tasks.ceph.cluster1.osd.1:Started 2026-02-18T06:19:59.804 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2026-02-18T06:19:59.804 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2026-02-18T06:19:59.806 INFO:tasks.ceph.cluster1.osd.2:Started 2026-02-18T06:19:59.807 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-18T06:19:59.911 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:19:59.911 INFO:teuthology.orchestra.run.trial031.stdout:{"epoch":5,"fsid":"c34c7f09-e418-4318-892f-b221bab3e02e","created":"2026-02-18T06:19:59.102474+0000","modified":"2026-02-18T06:19:59.789405+0000","last_up_change":"0.000000","last_in_change":"2026-02-18T06:19:59.789405+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"269a8c5d-3e3d-44c0-969c-686555b67a10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91fe9ddc-6f8a-4a7f-ae09-18bea6a1597b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d87b9373-e7c3-4a6e-b065-c5ee208d646f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:19:59.919 INFO:tasks.ceph.ceph_manager.cluster1:[] 2026-02-18T06:19:59.919 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-18T06:20:00.105 INFO:tasks.ceph.cluster1.osd.1.trial031.stderr:2026-02-18T06:20:00.102+0000 7f67c055f900 -1 Falling back to public interface 2026-02-18T06:20:00.106 INFO:tasks.ceph.cluster1.osd.0.trial031.stderr:2026-02-18T06:20:00.103+0000 7f6d767cf900 -1 Falling back to public interface 2026-02-18T06:20:00.115 INFO:tasks.ceph.cluster1.osd.2.trial031.stderr:2026-02-18T06:20:00.112+0000 7f6c9f3c6900 -1 Falling back to public interface 2026-02-18T06:20:00.145 INFO:tasks.ceph.cluster1.mgr.x.trial031.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-18T06:20:00.145 INFO:tasks.ceph.cluster1.mgr.x.trial031.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-18T06:20:00.145 INFO:tasks.ceph.cluster1.mgr.x.trial031.stderr: from numpy import show_config as show_numpy_config 2026-02-18T06:20:00.221 DEBUG:teuthology.orchestra.run.trial031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-18T06:20:00.485 INFO:teuthology.misc.health.trial031.stderr:2026-02-18T06:20:00.481+0000 7f1f59c94640 0 --2- 10.20.193.31:0/2661134934 >> [v2:10.20.193.31:3300/0,v1:10.20.193.31:6789/0] conn(0x7f1f54093530 0x7f1f54124d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-18T06:20:00.516 INFO:teuthology.misc.health.trial031.stdout: 2026-02-18T06:20:00.517 INFO:teuthology.misc.health.trial031.stdout:{"epoch":5,"fsid":"c34c7f09-e418-4318-892f-b221bab3e02e","created":"2026-02-18T06:19:59.102474+0000","modified":"2026-02-18T06:19:59.789405+0000","last_up_change":"0.000000","last_in_change":"2026-02-18T06:19:59.789405+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"269a8c5d-3e3d-44c0-969c-686555b67a10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"91fe9ddc-6f8a-4a7f-ae09-18bea6a1597b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d87b9373-e7c3-4a6e-b065-c5ee208d646f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:00.525 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-18T06:20:00.717 INFO:tasks.ceph.cluster1.osd.1.trial031.stderr:2026-02-18T06:20:00.714+0000 7f67c055f900 -1 osd.1 0 log_to_monitors true 2026-02-18T06:20:00.740 INFO:tasks.ceph.cluster1.osd.2.trial031.stderr:2026-02-18T06:20:00.737+0000 7f6c9f3c6900 -1 osd.2 0 log_to_monitors true 2026-02-18T06:20:00.756 INFO:tasks.ceph.cluster1.osd.0.trial031.stderr:2026-02-18T06:20:00.753+0000 7f6d767cf900 -1 osd.0 0 log_to_monitors true 2026-02-18T06:20:02.113 INFO:tasks.ceph.cluster1.osd.1.trial031.stderr:2026-02-18T06:20:02.109+0000 7f67bbcca640 -1 osd.1 0 waiting for initial osdmap 2026-02-18T06:20:02.113 INFO:tasks.ceph.cluster1.osd.2.trial031.stderr:2026-02-18T06:20:02.109+0000 7f6c99c35640 -1 osd.2 0 waiting for initial osdmap 2026-02-18T06:20:02.113 INFO:tasks.ceph.cluster1.osd.0.trial031.stderr:2026-02-18T06:20:02.110+0000 7f6d71f3c640 -1 osd.0 0 waiting for initial osdmap 2026-02-18T06:20:02.123 INFO:tasks.ceph.cluster1.osd.1.trial031.stderr:2026-02-18T06:20:02.120+0000 7f67b3ada640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-18T06:20:02.130 INFO:tasks.ceph.cluster1.osd.0.trial031.stderr:2026-02-18T06:20:02.127+0000 7f6d69d4c640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-18T06:20:02.130 INFO:tasks.ceph.cluster1.osd.2.trial031.stderr:2026-02-18T06:20:02.127+0000 7f6c91a45640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-18T06:20:02.532 INFO:tasks.ceph.cluster1.mgr.x.trial031.stderr:2026-02-18T06:20:02.529+0000 7f26e7de4640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-18T06:20:06.827 DEBUG:teuthology.orchestra.run.trial031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-18T06:20:07.001 INFO:teuthology.misc.health.trial031.stdout: 2026-02-18T06:20:07.001 INFO:teuthology.misc.health.trial031.stdout:{"epoch":11,"fsid":"c34c7f09-e418-4318-892f-b221bab3e02e","created":"2026-02-18T06:19:59.102474+0000","modified":"2026-02-18T06:20:06.532179+0000","last_up_change":"2026-02-18T06:20:03.109548+0000","last_in_change":"2026-02-18T06:19:59.789405+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:03.538981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"269a8c5d-3e3d-44c0-969c-686555b67a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6808","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6809","nonce":2229877145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6810","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6811","nonce":2229877145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6814","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6815","nonce":2229877145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6812","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6813","nonce":2229877145}]},"public_addr":"10.20.193.31:6809/2229877145","cluster_addr":"10.20.193.31:6811/2229877145","heartbeat_back_addr":"10.20.193.31:6815/2229877145","heartbeat_front_addr":"10.20.193.31:6813/2229877145","state":["exists","up"]},{"osd":1,"uuid":"91fe9ddc-6f8a-4a7f-ae09-18bea6a1597b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6800","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6801","nonce":1524508517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6802","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6803","nonce":1524508517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6806","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6807","nonce":1524508517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6804","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6805","nonce":1524508517}]},"public_addr":"10.20.193.31:6801/1524508517","cluster_addr":"10.20.193.31:6803/1524508517","heartbeat_back_addr":"10.20.193.31:6807/1524508517","heartbeat_front_addr":"10.20.193.31:6805/1524508517","state":["exists","up"]},{"osd":2,"uuid":"d87b9373-e7c3-4a6e-b065-c5ee208d646f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6816","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6817","nonce":1969377339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6818","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6819","nonce":1969377339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6822","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6823","nonce":1969377339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6820","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6821","nonce":1969377339}]},"public_addr":"10.20.193.31:6817/1969377339","cluster_addr":"10.20.193.31:6819/1969377339","heartbeat_back_addr":"10.20.193.31:6823/1969377339","heartbeat_front_addr":"10.20.193.31:6821/1969377339","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:07.009 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-18T06:20:07.009 INFO:tasks.ceph:Creating RBD pool 2026-02-18T06:20:07.010 DEBUG:teuthology.orchestra.run.trial031:> sudo ceph --cluster cluster1 osd pool create rbd 8 2026-02-18T06:20:07.537 INFO:teuthology.orchestra.run.trial031.stderr:pool 'rbd' created 2026-02-18T06:20:07.548 DEBUG:teuthology.orchestra.run.trial031:> rbd --cluster cluster1 pool init rbd 2026-02-18T06:20:10.550 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2026-02-18T06:20:10.551 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 config log 1 --format=json 2026-02-18T06:20:10.552 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-18T06:20:10.731 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:10.739 INFO:teuthology.orchestra.run.trial031.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-18T06:20:10.740 INFO:tasks.ceph_manager:config epoch is 1 2026-02-18T06:20:10.740 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2026-02-18T06:20:10.740 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2026-02-18T06:20:10.740 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2026-02-18T06:20:10.955 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:10.963 INFO:teuthology.orchestra.run.trial031.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6824","nonce":2918792010},{"type":"v1","addr":"10.20.193.31:6825","nonce":2918792010}]},"active_addr":"10.20.193.31:6825/2918792010","active_change":"2026-02-18T06:20:01.524855+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":4135675651}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":2834216653}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":1386684814}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":3132613383}]}]} 2026-02-18T06:20:10.966 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2026-02-18T06:20:10.967 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2026-02-18T06:20:10.967 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-18T06:20:11.146 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:11.146 INFO:teuthology.orchestra.run.trial031.stdout:{"epoch":15,"fsid":"c34c7f09-e418-4318-892f-b221bab3e02e","created":"2026-02-18T06:19:59.102474+0000","modified":"2026-02-18T06:20:10.540513+0000","last_up_change":"2026-02-18T06:20:03.109548+0000","last_in_change":"2026-02-18T06:19:59.789405+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:03.538981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-18T06:20:07.190513+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"269a8c5d-3e3d-44c0-969c-686555b67a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6808","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6809","nonce":2229877145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6810","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6811","nonce":2229877145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6814","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6815","nonce":2229877145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6812","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6813","nonce":2229877145}]},"public_addr":"10.20.193.31:6809/2229877145","cluster_addr":"10.20.193.31:6811/2229877145","heartbeat_back_addr":"10.20.193.31:6815/2229877145","heartbeat_front_addr":"10.20.193.31:6813/2229877145","state":["exists","up"]},{"osd":1,"uuid":"91fe9ddc-6f8a-4a7f-ae09-18bea6a1597b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6800","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6801","nonce":1524508517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6802","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6803","nonce":1524508517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6806","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6807","nonce":1524508517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6804","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6805","nonce":1524508517}]},"public_addr":"10.20.193.31:6801/1524508517","cluster_addr":"10.20.193.31:6803/1524508517","heartbeat_back_addr":"10.20.193.31:6807/1524508517","heartbeat_front_addr":"10.20.193.31:6805/1524508517","state":["exists","up"]},{"osd":2,"uuid":"d87b9373-e7c3-4a6e-b065-c5ee208d646f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6816","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6817","nonce":1969377339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6818","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6819","nonce":1969377339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6822","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6823","nonce":1969377339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6820","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6821","nonce":1969377339}]},"public_addr":"10.20.193.31:6817/1969377339","cluster_addr":"10.20.193.31:6819/1969377339","heartbeat_back_addr":"10.20.193.31:6823/1969377339","heartbeat_front_addr":"10.20.193.31:6821/1969377339","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:11.155 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2026-02-18T06:20:11.155 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-18T06:20:11.336 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:11.336 INFO:teuthology.orchestra.run.trial031.stdout:{"epoch":15,"fsid":"c34c7f09-e418-4318-892f-b221bab3e02e","created":"2026-02-18T06:19:59.102474+0000","modified":"2026-02-18T06:20:10.540513+0000","last_up_change":"2026-02-18T06:20:03.109548+0000","last_in_change":"2026-02-18T06:19:59.789405+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:03.538981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-18T06:20:07.190513+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"269a8c5d-3e3d-44c0-969c-686555b67a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6808","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6809","nonce":2229877145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6810","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6811","nonce":2229877145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6814","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6815","nonce":2229877145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6812","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6813","nonce":2229877145}]},"public_addr":"10.20.193.31:6809/2229877145","cluster_addr":"10.20.193.31:6811/2229877145","heartbeat_back_addr":"10.20.193.31:6815/2229877145","heartbeat_front_addr":"10.20.193.31:6813/2229877145","state":["exists","up"]},{"osd":1,"uuid":"91fe9ddc-6f8a-4a7f-ae09-18bea6a1597b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6800","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6801","nonce":1524508517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6802","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6803","nonce":1524508517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6806","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6807","nonce":1524508517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6804","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6805","nonce":1524508517}]},"public_addr":"10.20.193.31:6801/1524508517","cluster_addr":"10.20.193.31:6803/1524508517","heartbeat_back_addr":"10.20.193.31:6807/1524508517","heartbeat_front_addr":"10.20.193.31:6805/1524508517","state":["exists","up"]},{"osd":2,"uuid":"d87b9373-e7c3-4a6e-b065-c5ee208d646f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6816","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6817","nonce":1969377339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6818","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6819","nonce":1969377339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6822","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6823","nonce":1969377339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6820","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6821","nonce":1969377339}]},"public_addr":"10.20.193.31:6817/1969377339","cluster_addr":"10.20.193.31:6819/1969377339","heartbeat_back_addr":"10.20.193.31:6823/1969377339","heartbeat_front_addr":"10.20.193.31:6821/1969377339","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:11.346 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2026-02-18T06:20:11.346 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2026-02-18T06:20:11.347 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2026-02-18T06:20:11.435 INFO:teuthology.orchestra.run.trial031.stdout:34359738372 2026-02-18T06:20:11.435 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-18T06:20:11.436 INFO:teuthology.orchestra.run.trial031.stdout:34359738371 2026-02-18T06:20:11.437 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2026-02-18T06:20:11.438 INFO:teuthology.orchestra.run.trial031.stdout:34359738371 2026-02-18T06:20:11.438 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-18T06:20:11.657 INFO:teuthology.orchestra.run.trial031.stdout:34359738372 2026-02-18T06:20:11.658 INFO:teuthology.orchestra.run.trial031.stdout:34359738371 2026-02-18T06:20:11.664 INFO:teuthology.orchestra.run.trial031.stdout:34359738371 2026-02-18T06:20:11.668 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2026-02-18T06:20:11.668 DEBUG:teuthology.parallel:result is None 2026-02-18T06:20:11.669 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2026-02-18T06:20:11.669 DEBUG:teuthology.parallel:result is None 2026-02-18T06:20:11.674 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2026-02-18T06:20:11.675 DEBUG:teuthology.parallel:result is None 2026-02-18T06:20:11.675 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-18T06:20:11.675 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-18T06:20:11.893 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:11.894 INFO:teuthology.orchestra.run.trial031.stderr:dumped all 2026-02-18T06:20:11.902 INFO:teuthology.orchestra.run.trial031.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-18T06:20:11.529138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82424,"kb_used_data":1544,"kb_used_omap":23,"kb_used_meta":80808,"kb_avail":283033096,"statfs":{"total":289910292480,"available":289825890304,"internally_reserved":0,"allocated":1581056,"data_stored":1291807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24374,"internal_metadata":82747594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991927"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.545317+0000","last_change":"2026-02-18T06:20:10.545651+0000","last_active":"2026-02-18T06:20:10.545317+0000","last_peered":"2026-02-18T06:20:10.545317+0000","last_clean":"2026-02-18T06:20:10.545317+0000","last_became_active":"2026-02-18T06:20:08.542218+0000","last_became_peered":"2026-02-18T06:20:08.542218+0000","last_unstale":"2026-02-18T06:20:10.545317+0000","last_undegraded":"2026-02-18T06:20:10.545317+0000","last_fullsized":"2026-02-18T06:20:10.545317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:04:54.990306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095951000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.545035+0000","last_change":"2026-02-18T06:20:10.545246+0000","last_active":"2026-02-18T06:20:10.545035+0000","last_peered":"2026-02-18T06:20:10.545035+0000","last_clean":"2026-02-18T06:20:10.545035+0000","last_became_active":"2026-02-18T06:20:08.541842+0000","last_became_peered":"2026-02-18T06:20:08.541842+0000","last_unstale":"2026-02-18T06:20:10.545035+0000","last_undegraded":"2026-02-18T06:20:10.545035+0000","last_fullsized":"2026-02-18T06:20:10.545035+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:21:47.699026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059815900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.545101+0000","last_change":"2026-02-18T06:20:10.545338+0000","last_active":"2026-02-18T06:20:10.545101+0000","last_peered":"2026-02-18T06:20:10.545101+0000","last_clean":"2026-02-18T06:20:10.545101+0000","last_became_active":"2026-02-18T06:20:08.541791+0000","last_became_peered":"2026-02-18T06:20:08.541791+0000","last_unstale":"2026-02-18T06:20:10.545101+0000","last_undegraded":"2026-02-18T06:20:10.545101+0000","last_fullsized":"2026-02-18T06:20:10.545101+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:57:01.887084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068669399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.544875+0000","last_change":"2026-02-18T06:20:10.545063+0000","last_active":"2026-02-18T06:20:10.544875+0000","last_peered":"2026-02-18T06:20:10.544875+0000","last_clean":"2026-02-18T06:20:10.544875+0000","last_became_active":"2026-02-18T06:20:08.541914+0000","last_became_peered":"2026-02-18T06:20:08.541914+0000","last_unstale":"2026-02-18T06:20:10.544875+0000","last_undegraded":"2026-02-18T06:20:10.544875+0000","last_fullsized":"2026-02-18T06:20:10.544875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:35:47.720493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408427,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.543913+0000","last_change":"2026-02-18T06:20:10.543913+0000","last_active":"2026-02-18T06:20:10.543913+0000","last_peered":"2026-02-18T06:20:10.543913+0000","last_clean":"2026-02-18T06:20:10.543913+0000","last_became_active":"2026-02-18T06:20:08.541913+0000","last_became_peered":"2026-02-18T06:20:08.541913+0000","last_unstale":"2026-02-18T06:20:10.543913+0000","last_undegraded":"2026-02-18T06:20:10.543913+0000","last_fullsized":"2026-02-18T06:20:10.543913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:01:29.992866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077138100000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.739257+0000","last_change":"2026-02-18T06:20:10.739368+0000","last_active":"2026-02-18T06:20:10.739257+0000","last_peered":"2026-02-18T06:20:10.739257+0000","last_clean":"2026-02-18T06:20:10.739257+0000","last_became_active":"2026-02-18T06:20:08.541846+0000","last_became_peered":"2026-02-18T06:20:08.541846+0000","last_unstale":"2026-02-18T06:20:10.739257+0000","last_undegraded":"2026-02-18T06:20:10.739257+0000","last_fullsized":"2026-02-18T06:20:10.739257+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:19:18.113602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000219657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.739238+0000","last_change":"2026-02-18T06:20:10.739337+0000","last_active":"2026-02-18T06:20:10.739238+0000","last_peered":"2026-02-18T06:20:10.739238+0000","last_clean":"2026-02-18T06:20:10.739238+0000","last_became_active":"2026-02-18T06:20:08.541956+0000","last_became_peered":"2026-02-18T06:20:08.541956+0000","last_unstale":"2026-02-18T06:20:10.739238+0000","last_undegraded":"2026-02-18T06:20:10.739238+0000","last_fullsized":"2026-02-18T06:20:10.739238+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:06:02.012849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023866299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.544828+0000","last_change":"2026-02-18T06:20:10.544993+0000","last_active":"2026-02-18T06:20:10.544828+0000","last_peered":"2026-02-18T06:20:10.544828+0000","last_clean":"2026-02-18T06:20:10.544828+0000","last_became_active":"2026-02-18T06:20:08.541386+0000","last_became_peered":"2026-02-18T06:20:08.541386+0000","last_unstale":"2026-02-18T06:20:10.544828+0000","last_undegraded":"2026-02-18T06:20:10.544828+0000","last_fullsized":"2026-02-18T06:20:10.544828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:11:31.968149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034991900000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":110,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.544510+0000","last_change":"2026-02-18T06:20:05.694236+0000","last_active":"2026-02-18T06:20:10.544510+0000","last_peered":"2026-02-18T06:20:10.544510+0000","last_clean":"2026-02-18T06:20:10.544510+0000","last_became_active":"2026-02-18T06:20:05.693589+0000","last_became_peered":"2026-02-18T06:20:05.693589+0000","last_unstale":"2026-02-18T06:20:10.544510+0000","last_undegraded":"2026-02-18T06:20:10.544510+0000","last_fullsized":"2026-02-18T06:20:10.544510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_clean_scrub_stamp":"2026-02-18T06:20:04.531394+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:48:22.870932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":120,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":122880,"data_stored":34635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7475,"internal_metadata":27583181},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8774,"internal_metadata":27581882},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:20:11.903 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-18T06:20:12.080 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:12.080 INFO:teuthology.orchestra.run.trial031.stderr:dumped all 2026-02-18T06:20:12.088 INFO:teuthology.orchestra.run.trial031.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-18T06:20:11.529138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82424,"kb_used_data":1544,"kb_used_omap":23,"kb_used_meta":80808,"kb_avail":283033096,"statfs":{"total":289910292480,"available":289825890304,"internally_reserved":0,"allocated":1581056,"data_stored":1291807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24374,"internal_metadata":82747594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991927"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.545317+0000","last_change":"2026-02-18T06:20:10.545651+0000","last_active":"2026-02-18T06:20:10.545317+0000","last_peered":"2026-02-18T06:20:10.545317+0000","last_clean":"2026-02-18T06:20:10.545317+0000","last_became_active":"2026-02-18T06:20:08.542218+0000","last_became_peered":"2026-02-18T06:20:08.542218+0000","last_unstale":"2026-02-18T06:20:10.545317+0000","last_undegraded":"2026-02-18T06:20:10.545317+0000","last_fullsized":"2026-02-18T06:20:10.545317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:04:54.990306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095951000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.545035+0000","last_change":"2026-02-18T06:20:10.545246+0000","last_active":"2026-02-18T06:20:10.545035+0000","last_peered":"2026-02-18T06:20:10.545035+0000","last_clean":"2026-02-18T06:20:10.545035+0000","last_became_active":"2026-02-18T06:20:08.541842+0000","last_became_peered":"2026-02-18T06:20:08.541842+0000","last_unstale":"2026-02-18T06:20:10.545035+0000","last_undegraded":"2026-02-18T06:20:10.545035+0000","last_fullsized":"2026-02-18T06:20:10.545035+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:21:47.699026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059815900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.545101+0000","last_change":"2026-02-18T06:20:10.545338+0000","last_active":"2026-02-18T06:20:10.545101+0000","last_peered":"2026-02-18T06:20:10.545101+0000","last_clean":"2026-02-18T06:20:10.545101+0000","last_became_active":"2026-02-18T06:20:08.541791+0000","last_became_peered":"2026-02-18T06:20:08.541791+0000","last_unstale":"2026-02-18T06:20:10.545101+0000","last_undegraded":"2026-02-18T06:20:10.545101+0000","last_fullsized":"2026-02-18T06:20:10.545101+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:57:01.887084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068669399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.544875+0000","last_change":"2026-02-18T06:20:10.545063+0000","last_active":"2026-02-18T06:20:10.544875+0000","last_peered":"2026-02-18T06:20:10.544875+0000","last_clean":"2026-02-18T06:20:10.544875+0000","last_became_active":"2026-02-18T06:20:08.541914+0000","last_became_peered":"2026-02-18T06:20:08.541914+0000","last_unstale":"2026-02-18T06:20:10.544875+0000","last_undegraded":"2026-02-18T06:20:10.544875+0000","last_fullsized":"2026-02-18T06:20:10.544875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:35:47.720493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408427,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.543913+0000","last_change":"2026-02-18T06:20:10.543913+0000","last_active":"2026-02-18T06:20:10.543913+0000","last_peered":"2026-02-18T06:20:10.543913+0000","last_clean":"2026-02-18T06:20:10.543913+0000","last_became_active":"2026-02-18T06:20:08.541913+0000","last_became_peered":"2026-02-18T06:20:08.541913+0000","last_unstale":"2026-02-18T06:20:10.543913+0000","last_undegraded":"2026-02-18T06:20:10.543913+0000","last_fullsized":"2026-02-18T06:20:10.543913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:01:29.992866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077138100000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.739257+0000","last_change":"2026-02-18T06:20:10.739368+0000","last_active":"2026-02-18T06:20:10.739257+0000","last_peered":"2026-02-18T06:20:10.739257+0000","last_clean":"2026-02-18T06:20:10.739257+0000","last_became_active":"2026-02-18T06:20:08.541846+0000","last_became_peered":"2026-02-18T06:20:08.541846+0000","last_unstale":"2026-02-18T06:20:10.739257+0000","last_undegraded":"2026-02-18T06:20:10.739257+0000","last_fullsized":"2026-02-18T06:20:10.739257+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:19:18.113602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000219657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.739238+0000","last_change":"2026-02-18T06:20:10.739337+0000","last_active":"2026-02-18T06:20:10.739238+0000","last_peered":"2026-02-18T06:20:10.739238+0000","last_clean":"2026-02-18T06:20:10.739238+0000","last_became_active":"2026-02-18T06:20:08.541956+0000","last_became_peered":"2026-02-18T06:20:08.541956+0000","last_unstale":"2026-02-18T06:20:10.739238+0000","last_undegraded":"2026-02-18T06:20:10.739238+0000","last_fullsized":"2026-02-18T06:20:10.739238+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:06:02.012849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023866299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.544828+0000","last_change":"2026-02-18T06:20:10.544993+0000","last_active":"2026-02-18T06:20:10.544828+0000","last_peered":"2026-02-18T06:20:10.544828+0000","last_clean":"2026-02-18T06:20:10.544828+0000","last_became_active":"2026-02-18T06:20:08.541386+0000","last_became_peered":"2026-02-18T06:20:08.541386+0000","last_unstale":"2026-02-18T06:20:10.544828+0000","last_undegraded":"2026-02-18T06:20:10.544828+0000","last_fullsized":"2026-02-18T06:20:10.544828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:11:31.968149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034991900000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":110,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:10.544510+0000","last_change":"2026-02-18T06:20:05.694236+0000","last_active":"2026-02-18T06:20:10.544510+0000","last_peered":"2026-02-18T06:20:10.544510+0000","last_clean":"2026-02-18T06:20:10.544510+0000","last_became_active":"2026-02-18T06:20:05.693589+0000","last_became_peered":"2026-02-18T06:20:05.693589+0000","last_unstale":"2026-02-18T06:20:10.544510+0000","last_undegraded":"2026-02-18T06:20:10.544510+0000","last_fullsized":"2026-02-18T06:20:10.544510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_clean_scrub_stamp":"2026-02-18T06:20:04.531394+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:48:22.870932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":120,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":122880,"data_stored":34635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7475,"internal_metadata":27583181},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8774,"internal_metadata":27581882},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:20:12.089 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-18T06:20:12.090 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2026-02-18T06:20:12.090 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2026-02-18T06:20:12.090 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2026-02-18T06:20:12.296 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:12.296 INFO:teuthology.orchestra.run.trial031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-18T06:20:12.305 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2026-02-18T06:20:12.305 INFO:teuthology.run_tasks:Running task ceph... 2026-02-18T06:20:12.320 INFO:tasks.ceph:Creating ceph cluster cluster2... 2026-02-18T06:20:12.320 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-18T06:20:12.320 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885', 'branch': '20.2.1_2.17.26-baseline_2', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-policy-none}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '54885', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror.sh workunit to test the rbd-mirror daemon in journal mode'}], 'name': 'yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': '20.2.1_2.17.26-baseline_2'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': '20.2.1_2.17.26-baseline_2', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7878, 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': '20.2.1_2.17.26-baseline_2', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed', 'targets': {'trial031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHRNW7A2DvvWlKHRZNI6CLZ97SvCZ/eAtdF852Rm9tFeUL7CxpN5UeNDcxfpYyTzxAc0h0SLbexV5u04UgbTlx4=', 'trial130.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEwrfL57bUfT56dxS9dPm4meFgSKQcTf5bVd/MCniJ+6npZE+yMJCdieVstdiz0p0rICGb+5W5V0e2I6sZx+XsM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-17_21:26:40', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682'} 2026-02-18T06:20:12.321 DEBUG:teuthology.orchestra.run.trial031:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-18T06:20:12.324 DEBUG:teuthology.orchestra.run.trial130:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-18T06:20:12.341 DEBUG:teuthology.orchestra.run.trial031:> sudo install -d -m0777 -- /var/run/ceph 2026-02-18T06:20:12.380 DEBUG:teuthology.orchestra.run.trial130:> sudo install -d -m0777 -- /var/run/ceph 2026-02-18T06:20:12.407 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:12.407 DEBUG:teuthology.orchestra.run.trial130:> dd if=/scratch_devs of=/dev/stdout 2026-02-18T06:20:12.463 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-18T06:20:12.463 DEBUG:teuthology.orchestra.run.trial130:> stat /dev/vg_nvme/lv_1 2026-02-18T06:20:12.521 INFO:teuthology.orchestra.run.trial130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-18T06:20:12.521 INFO:teuthology.orchestra.run.trial130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:20:12.521 INFO:teuthology.orchestra.run.trial130.stdout:Device: 6h/6d Inode: 1313 Links: 1 2026-02-18T06:20:12.522 INFO:teuthology.orchestra.run.trial130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:20:12.522 INFO:teuthology.orchestra.run.trial130.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:20:12.522 INFO:teuthology.orchestra.run.trial130.stdout:Access: 2026-02-18 06:19:43.332656865 +0000 2026-02-18T06:20:12.522 INFO:teuthology.orchestra.run.trial130.stdout:Modify: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:12.522 INFO:teuthology.orchestra.run.trial130.stdout:Change: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:12.522 INFO:teuthology.orchestra.run.trial130.stdout: Birth: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:12.522 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-18T06:20:12.586 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records in 2026-02-18T06:20:12.586 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records out 2026-02-18T06:20:12.586 INFO:teuthology.orchestra.run.trial130.stderr:512 bytes copied, 0.000155278 s, 3.3 MB/s 2026-02-18T06:20:12.587 DEBUG:teuthology.orchestra.run.trial130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-18T06:20:12.646 DEBUG:teuthology.orchestra.run.trial130:> stat /dev/vg_nvme/lv_2 2026-02-18T06:20:12.702 INFO:teuthology.orchestra.run.trial130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-18T06:20:12.702 INFO:teuthology.orchestra.run.trial130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:20:12.703 INFO:teuthology.orchestra.run.trial130.stdout:Device: 6h/6d Inode: 1322 Links: 1 2026-02-18T06:20:12.703 INFO:teuthology.orchestra.run.trial130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:20:12.703 INFO:teuthology.orchestra.run.trial130.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:20:12.703 INFO:teuthology.orchestra.run.trial130.stdout:Access: 2026-02-18 06:19:43.332656865 +0000 2026-02-18T06:20:12.703 INFO:teuthology.orchestra.run.trial130.stdout:Modify: 2026-02-18 06:19:04.412612974 +0000 2026-02-18T06:20:12.703 INFO:teuthology.orchestra.run.trial130.stdout:Change: 2026-02-18 06:19:04.412612974 +0000 2026-02-18T06:20:12.703 INFO:teuthology.orchestra.run.trial130.stdout: Birth: 2026-02-18 06:19:04.412612974 +0000 2026-02-18T06:20:12.703 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-18T06:20:12.764 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records in 2026-02-18T06:20:12.765 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records out 2026-02-18T06:20:12.765 INFO:teuthology.orchestra.run.trial130.stderr:512 bytes copied, 0.000144548 s, 3.5 MB/s 2026-02-18T06:20:12.766 DEBUG:teuthology.orchestra.run.trial130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-18T06:20:12.823 DEBUG:teuthology.orchestra.run.trial130:> stat /dev/vg_nvme/lv_3 2026-02-18T06:20:12.879 INFO:teuthology.orchestra.run.trial130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-18T06:20:12.879 INFO:teuthology.orchestra.run.trial130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:20:12.879 INFO:teuthology.orchestra.run.trial130.stdout:Device: 6h/6d Inode: 1312 Links: 1 2026-02-18T06:20:12.880 INFO:teuthology.orchestra.run.trial130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:20:12.880 INFO:teuthology.orchestra.run.trial130.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:20:12.880 INFO:teuthology.orchestra.run.trial130.stdout:Access: 2026-02-18 06:19:43.332656865 +0000 2026-02-18T06:20:12.880 INFO:teuthology.orchestra.run.trial130.stdout:Modify: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:12.880 INFO:teuthology.orchestra.run.trial130.stdout:Change: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:12.880 INFO:teuthology.orchestra.run.trial130.stdout: Birth: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:12.880 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-18T06:20:12.941 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records in 2026-02-18T06:20:12.942 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records out 2026-02-18T06:20:12.942 INFO:teuthology.orchestra.run.trial130.stderr:512 bytes copied, 0.000154277 s, 3.3 MB/s 2026-02-18T06:20:12.943 DEBUG:teuthology.orchestra.run.trial130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-18T06:20:13.001 DEBUG:teuthology.orchestra.run.trial130:> stat /dev/vg_nvme/lv_4 2026-02-18T06:20:13.059 INFO:teuthology.orchestra.run.trial130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout:Device: 6h/6d Inode: 1298 Links: 1 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout:Context: system_u:object_r:device_t:s0 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout:Access: 2026-02-18 06:19:43.333656841 +0000 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout:Modify: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout:Change: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:13.060 INFO:teuthology.orchestra.run.trial130.stdout: Birth: 2026-02-18 06:19:04.411612998 +0000 2026-02-18T06:20:13.061 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-18T06:20:13.122 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records in 2026-02-18T06:20:13.122 INFO:teuthology.orchestra.run.trial130.stderr:1+0 records out 2026-02-18T06:20:13.122 INFO:teuthology.orchestra.run.trial130.stderr:512 bytes copied, 0.000141403 s, 3.6 MB/s 2026-02-18T06:20:13.123 DEBUG:teuthology.orchestra.run.trial130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-18T06:20:13.182 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:20:13.182 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial130.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-18T06:20:13.182 INFO:tasks.ceph:Generating config... 2026-02-18T06:20:13.184 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client] log to stderr = False 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.185 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.186 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.186 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.186 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.186 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.186 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.186 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.186 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.186 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-18T06:20:13.186 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-18T06:20:13.186 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-18T06:20:13.186 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-18T06:20:13.186 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-18T06:20:13.187 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-18T06:20:13.187 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-18T06:20:13.187 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-18T06:20:13.187 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-18T06:20:13.188 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-18T06:20:13.188 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-18T06:20:13.188 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-18T06:20:13.188 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-18T06:20:13.188 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-18T06:20:13.188 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-18T06:20:13.188 INFO:tasks.ceph:Setting up cluster2.mon.a... 2026-02-18T06:20:13.188 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2026-02-18T06:20:13.256 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.keyring 2026-02-18T06:20:13.259 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2026-02-18T06:20:13.336 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-18T06:20:13.400 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '10.20.193.130')] 2026-02-18T06:20:13.401 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.130', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-18T06:20:13.402 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:13.402 DEBUG:teuthology.orchestra.run.trial130:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-18T06:20:13.456 DEBUG:teuthology.orchestra.run.trial130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.130 --print /home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:13.528 INFO:teuthology.orchestra.run.trial130.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:monmaptool: generated fsid 59a365c0-617d-43e5-831e-67d120cf5f4b 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:setting min_mon_release = tentacle 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:epoch 0 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:fsid 59a365c0-617d-43e5-831e-67d120cf5f4b 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:last_changed 2026-02-18T06:20:13.529136+0000 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:created 2026-02-18T06:20:13.529136+0000 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:min_mon_release 20 (tentacle) 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:election_strategy: 1 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:0: [v2:10.20.193.130:3300/0,v1:10.20.193.130:6789/0] mon.a 2026-02-18T06:20:13.529 INFO:teuthology.orchestra.run.trial130.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2026-02-18T06:20:13.530 DEBUG:teuthology.orchestra.run.trial130:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-18T06:20:13.587 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 59a365c0-617d-43e5-831e-67d120cf5f4b... 2026-02-18T06:20:13.605 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-18T06:20:13.608 DEBUG:teuthology.orchestra.run.trial130:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-18T06:20:13.647 INFO:teuthology.orchestra.run.trial031.stdout:[global] 2026-02-18T06:20:13.647 INFO:teuthology.orchestra.run.trial031.stdout: chdir = "" 2026-02-18T06:20:13.647 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-18T06:20:13.647 INFO:teuthology.orchestra.run.trial031.stdout: auth supported = cephx 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: filestore xattr use omap = true 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: mon clock drift allowed = 1.000 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: osd crush chooseleaf type = 0 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: auth debug = true 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: ms die on old message = true 2026-02-18T06:20:13.648 INFO:teuthology.orchestra.run.trial031.stdout: ms die on bug = true 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: mon pg warn max object skew = 0 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: # disable pg_autoscaler by default for new pools 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: osd pool default size = 2 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.649 INFO:teuthology.orchestra.run.trial031.stdout: mon osd allow primary affinity = true 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon osd allow pg remap = true 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on legacy crush tunables = false 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on crush straw calc version zero = false 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on no sortbitwise = false 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on osd down out interval zero = false 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on too few osds = false 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-18T06:20:13.650 INFO:teuthology.orchestra.run.trial031.stdout: mon_allow_pool_size_one = true 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: osd default data pool replay window = 5 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: mon allow pool delete = true 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.651 INFO:teuthology.orchestra.run.trial031.stdout: mon cluster log file level = debug 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: debug asserts on shutdown = true 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: mon health detail to clog = false 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: mon host = 10.20.193.130 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: mon client directed command retry = 5 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: mon warn on pool no app = False 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: ms inject socket failures = 5000 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: fsid = 59a365c0-617d-43e5-831e-67d120cf5f4b 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout:[osd] 2026-02-18T06:20:13.652 INFO:teuthology.orchestra.run.trial031.stdout: osd journal size = 100 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd scrub load threshold = 5.0 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd scrub max interval = 600 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd mclock profile = high_recovery_ops 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd mclock skip benchmark = true 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd recover clone overlap = true 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd recovery max chunk = 1048576 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd debug shutdown = true 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd debug op order = true 2026-02-18T06:20:13.653 INFO:teuthology.orchestra.run.trial031.stdout: osd debug verify stray on activate = true 2026-02-18T06:20:13.654 INFO:teuthology.orchestra.run.trial031.stdout: osd debug trim objects = true 2026-02-18T06:20:13.654 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.654 INFO:teuthology.orchestra.run.trial031.stdout: osd open classes on start = true 2026-02-18T06:20:13.654 INFO:teuthology.orchestra.run.trial031.stdout: osd debug pg log writeout = true 2026-02-18T06:20:13.654 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.654 INFO:teuthology.orchestra.run.trial031.stdout: osd deep scrub update digest min age = 30 2026-02-18T06:20:13.654 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: osd map max advance = 10 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: journal zero on create = true 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: filestore ondisk finisher threads = 3 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: filestore apply finisher threads = 3 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: bdev debug aio = true 2026-02-18T06:20:13.655 INFO:teuthology.orchestra.run.trial031.stdout: osd debug misdirected ops = true 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: bdev async discard = True 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: bdev enable discard = True 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: bluestore allocator = bitmap 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: bluestore block size = 96636764160 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: bluestore fsck on mount = True 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: debug bluefs = 1/20 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: debug bluestore = 1/20 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: debug ms = 1 2026-02-18T06:20:13.656 INFO:teuthology.orchestra.run.trial031.stdout: debug osd = 20 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: debug rocksdb = 4/10 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: mon osd backfillfull_ratio = 0.85 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: mon osd full ratio = 0.9 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: mon osd nearfull ratio = 0.8 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: osd failsafe full ratio = 0.95 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: osd objectstore = bluestore 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout:[mgr] 2026-02-18T06:20:13.657 INFO:teuthology.orchestra.run.trial031.stdout: debug ms = 1 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: debug mgr = 20 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: debug mon = 20 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: debug auth = 20 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: mgr/telemetry/nag = false 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout:[mon] 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: debug ms = 1 2026-02-18T06:20:13.658 INFO:teuthology.orchestra.run.trial031.stdout: debug mon = 20 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: debug paxos = 20 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: debug auth = 20 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: mon data avail warn = 5 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: mon mgr mkfs grace = 240 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: mon osd reporter subtree level = osd 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: mon osd prime pg temp = true 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-18T06:20:13.659 INFO:teuthology.orchestra.run.trial031.stdout: auth mon ticket ttl = 660 # 11m 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: auth service ticket ttl = 240 # 4m 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: # don't complain about insecure global_id in the test suite 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: # 1m isn't quite enough 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: mon_down_mkfs_grace = 2m 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: mon_warn_on_filestore_osds = false 2026-02-18T06:20:13.660 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout:[client] 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: rgw cache enabled = true 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: rgw enable ops log = true 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: rgw enable usage log = true 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: debug rbd = 20 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: debug rbd_mirror = 15 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: log to stderr = False 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: rbd default features = 125 2026-02-18T06:20:13.661 INFO:teuthology.orchestra.run.trial031.stdout: rbd mirror image policy type = none 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout:[mon.a] 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.0] 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.1] 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.2] 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.662 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.3] 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.4] 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.5] 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout:[client.mirror.6] 2026-02-18T06:20:13.663 INFO:teuthology.orchestra.run.trial031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.664 INFO:teuthology.orchestra.run.trial031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout:[global] 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout: chdir = "" 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout: auth supported = cephx 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout: filestore xattr use omap = true 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.674 INFO:teuthology.orchestra.run.trial130.stdout: mon clock drift allowed = 1.000 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: osd crush chooseleaf type = 0 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: auth debug = true 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: ms die on old message = true 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: ms die on bug = true 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: mon pg warn max object skew = 0 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.675 INFO:teuthology.orchestra.run.trial130.stdout: # disable pg_autoscaler by default for new pools 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: osd pool default size = 2 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon osd allow primary affinity = true 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon osd allow pg remap = true 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on legacy crush tunables = false 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on crush straw calc version zero = false 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on no sortbitwise = false 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on osd down out interval zero = false 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on too few osds = false 2026-02-18T06:20:13.676 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: mon_allow_pool_size_one = true 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: osd default data pool replay window = 5 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: mon allow pool delete = true 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: mon cluster log file level = debug 2026-02-18T06:20:13.677 INFO:teuthology.orchestra.run.trial130.stdout: debug asserts on shutdown = true 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: mon health detail to clog = false 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: mon host = 10.20.193.130 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: mon client directed command retry = 5 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: mon warn on pool no app = False 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: ms inject socket failures = 5000 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: fsid = 59a365c0-617d-43e5-831e-67d120cf5f4b 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout:[osd] 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: osd journal size = 100 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: osd scrub load threshold = 5.0 2026-02-18T06:20:13.678 INFO:teuthology.orchestra.run.trial130.stdout: osd scrub max interval = 600 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd mclock profile = high_recovery_ops 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd mclock skip benchmark = true 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd recover clone overlap = true 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd recovery max chunk = 1048576 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd debug shutdown = true 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd debug op order = true 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd debug verify stray on activate = true 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd debug trim objects = true 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.679 INFO:teuthology.orchestra.run.trial130.stdout: osd open classes on start = true 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: osd debug pg log writeout = true 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: osd deep scrub update digest min age = 30 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: osd map max advance = 10 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: journal zero on create = true 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: filestore ondisk finisher threads = 3 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: filestore apply finisher threads = 3 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: bdev debug aio = true 2026-02-18T06:20:13.680 INFO:teuthology.orchestra.run.trial130.stdout: osd debug misdirected ops = true 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: bdev async discard = True 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: bdev enable discard = True 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: bluestore allocator = bitmap 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: bluestore block size = 96636764160 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: bluestore fsck on mount = True 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: debug bluefs = 1/20 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: debug bluestore = 1/20 2026-02-18T06:20:13.681 INFO:teuthology.orchestra.run.trial130.stdout: debug ms = 1 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: debug osd = 20 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: debug rocksdb = 4/10 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: mon osd backfillfull_ratio = 0.85 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: mon osd full ratio = 0.9 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: mon osd nearfull ratio = 0.8 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: osd failsafe full ratio = 0.95 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: osd objectstore = bluestore 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout:[mgr] 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: debug ms = 1 2026-02-18T06:20:13.682 INFO:teuthology.orchestra.run.trial130.stdout: debug mgr = 20 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: debug mon = 20 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: debug auth = 20 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: mgr/telemetry/nag = false 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout:[mon] 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: debug ms = 1 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: debug mon = 20 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: debug paxos = 20 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: debug auth = 20 2026-02-18T06:20:13.683 INFO:teuthology.orchestra.run.trial130.stdout: mon data avail warn = 5 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: mon mgr mkfs grace = 240 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min pgs per osd = 4 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: mon osd reporter subtree level = osd 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: mon osd prime pg temp = true 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: mon reweight min bytes per osd = 10 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: auth mon ticket ttl = 660 # 11m 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: auth service ticket ttl = 240 # 4m 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: # don't complain about insecure global_id in the test suite 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-18T06:20:13.684 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: # 1m isn't quite enough 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: mon_down_mkfs_grace = 2m 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: mon_warn_on_filestore_osds = false 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout:[client] 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: rgw cache enabled = true 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: rgw enable ops log = true 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: rgw enable usage log = true 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-18T06:20:13.685 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: debug rbd = 20 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: debug rbd_mirror = 15 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: log to stderr = False 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: rbd default features = 125 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: rbd mirror image policy type = none 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout:[mon.a] 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.0] 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.1] 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.686 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.2] 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.3] 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.4] 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.5] 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.6] 2026-02-18T06:20:13.687 INFO:teuthology.orchestra.run.trial130.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-18T06:20:13.688 INFO:teuthology.orchestra.run.trial130.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-18T06:20:13.689 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2026-02-18T06:20:13.689 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2026-02-18T06:20:13.773 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-18T06:20:13.773 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:13.773 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2026-02-18T06:20:13.830 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:13.830 DEBUG:teuthology.orchestra.run.trial130:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2026-02-18T06:20:13.886 INFO:tasks.ceph:Sending monmap to node ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:20:13.886 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:20:13.886 DEBUG:teuthology.orchestra.run.trial031:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-18T06:20:13.886 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-18T06:20:13.920 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:20:13.920 DEBUG:teuthology.orchestra.run.trial031:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:13.976 INFO:tasks.ceph:Sending monmap to node ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:20:13.976 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:13.976 DEBUG:teuthology.orchestra.run.trial130:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-18T06:20:13.976 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-18T06:20:14.010 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:14.010 DEBUG:teuthology.orchestra.run.trial130:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:14.067 INFO:tasks.ceph:Setting up mon nodes... 2026-02-18T06:20:14.068 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-18T06:20:14.068 DEBUG:teuthology.orchestra.run.trial031:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-18T06:20:14.112 INFO:teuthology.orchestra.run.trial031.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-18T06:20:14.115 INFO:tasks.ceph:Setting up mds nodes... 2026-02-18T06:20:14.115 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-18T06:20:14.115 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2026-02-18T06:20:14.149 INFO:teuthology.orchestra.run.trial031.stdout:creating /etc/ceph/cluster2.client.0.keyring 2026-02-18T06:20:14.160 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2026-02-18T06:20:14.196 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2026-02-18T06:20:14.207 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-18T06:20:14.282 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-18T06:20:14.292 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-18T06:20:14.368 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-18T06:20:14.380 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-18T06:20:14.452 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-18T06:20:14.464 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-18T06:20:14.539 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-18T06:20:14.549 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-18T06:20:14.623 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-18T06:20:14.634 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-18T06:20:14.709 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-18T06:20:14.720 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-18T06:20:14.795 INFO:teuthology.orchestra.run.trial130.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-18T06:20:14.805 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-18T06:20:14.805 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@trial130.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-18T06:20:14.806 DEBUG:teuthology.orchestra.run.trial130:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2026-02-18T06:20:14.870 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:20:14.870 INFO:tasks.ceph:role: cluster2.osd.0 2026-02-18T06:20:14.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:20:14.871 DEBUG:teuthology.orchestra.run.trial130:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-18T06:20:14.935 INFO:teuthology.orchestra.run.trial130.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-18T06:20:14.935 INFO:teuthology.orchestra.run.trial130.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-18T06:20:14.935 INFO:teuthology.orchestra.run.trial130.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-18T06:20:14.936 INFO:teuthology.orchestra.run.trial130.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-18T06:20:14.936 INFO:teuthology.orchestra.run.trial130.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-18T06:20:14.936 INFO:teuthology.orchestra.run.trial130.stdout: = sunit=0 swidth=0 blks 2026-02-18T06:20:14.936 INFO:teuthology.orchestra.run.trial130.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-18T06:20:14.936 INFO:teuthology.orchestra.run.trial130.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-18T06:20:14.936 INFO:teuthology.orchestra.run.trial130.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-18T06:20:14.936 INFO:teuthology.orchestra.run.trial130.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-18T06:20:15.511 INFO:teuthology.orchestra.run.trial130.stdout:Discarding blocks...Done. 2026-02-18T06:20:15.513 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial130.front.sepia.ceph.com -o noatime 2026-02-18T06:20:15.513 DEBUG:teuthology.orchestra.run.trial130:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2026-02-18T06:20:15.581 DEBUG:teuthology.orchestra.run.trial130:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2026-02-18T06:20:15.648 DEBUG:teuthology.orchestra.run.trial130:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2026-02-18T06:20:15.711 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:20:15.711 INFO:tasks.ceph:role: cluster2.osd.1 2026-02-18T06:20:15.711 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:20:15.711 DEBUG:teuthology.orchestra.run.trial130:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-18T06:20:15.776 INFO:teuthology.orchestra.run.trial130.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-18T06:20:15.776 INFO:teuthology.orchestra.run.trial130.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-18T06:20:15.776 INFO:teuthology.orchestra.run.trial130.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-18T06:20:15.776 INFO:teuthology.orchestra.run.trial130.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-18T06:20:15.777 INFO:teuthology.orchestra.run.trial130.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-18T06:20:15.777 INFO:teuthology.orchestra.run.trial130.stdout: = sunit=0 swidth=0 blks 2026-02-18T06:20:15.777 INFO:teuthology.orchestra.run.trial130.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-18T06:20:15.777 INFO:teuthology.orchestra.run.trial130.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-18T06:20:15.777 INFO:teuthology.orchestra.run.trial130.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-18T06:20:15.777 INFO:teuthology.orchestra.run.trial130.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-18T06:20:16.334 INFO:teuthology.orchestra.run.trial130.stdout:Discarding blocks...Done. 2026-02-18T06:20:16.336 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial130.front.sepia.ceph.com -o noatime 2026-02-18T06:20:16.336 DEBUG:teuthology.orchestra.run.trial130:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2026-02-18T06:20:16.406 DEBUG:teuthology.orchestra.run.trial130:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2026-02-18T06:20:16.474 DEBUG:teuthology.orchestra.run.trial130:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2026-02-18T06:20:16.538 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-18T06:20:16.538 INFO:tasks.ceph:role: cluster2.osd.2 2026-02-18T06:20:16.538 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:20:16.538 DEBUG:teuthology.orchestra.run.trial130:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-18T06:20:16.605 INFO:teuthology.orchestra.run.trial130.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-18T06:20:16.605 INFO:teuthology.orchestra.run.trial130.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-18T06:20:16.605 INFO:teuthology.orchestra.run.trial130.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-18T06:20:16.605 INFO:teuthology.orchestra.run.trial130.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-18T06:20:16.605 INFO:teuthology.orchestra.run.trial130.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-18T06:20:16.605 INFO:teuthology.orchestra.run.trial130.stdout: = sunit=0 swidth=0 blks 2026-02-18T06:20:16.605 INFO:teuthology.orchestra.run.trial130.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-18T06:20:16.606 INFO:teuthology.orchestra.run.trial130.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-18T06:20:16.606 INFO:teuthology.orchestra.run.trial130.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-18T06:20:16.606 INFO:teuthology.orchestra.run.trial130.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-18T06:20:17.169 INFO:teuthology.orchestra.run.trial130.stdout:Discarding blocks...Done. 2026-02-18T06:20:17.171 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial130.front.sepia.ceph.com -o noatime 2026-02-18T06:20:17.171 DEBUG:teuthology.orchestra.run.trial130:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2026-02-18T06:20:17.238 DEBUG:teuthology.orchestra.run.trial130:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2026-02-18T06:20:17.305 DEBUG:teuthology.orchestra.run.trial130:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:17.385 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:17.385+0000 7f0ad0bc1900 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2026-02-18T06:20:17.386 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:17.385+0000 7f0ad0bc1900 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2026-02-18T06:20:17.386 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:17.385+0000 7f0ad0bc1900 -1 bdev(0x557f5106f800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2026-02-18T06:20:17.386 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:17.385+0000 7f0ad0bc1900 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2026-02-18T06:20:17.864 DEBUG:teuthology.orchestra.run.trial130:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2026-02-18T06:20:17.929 DEBUG:teuthology.orchestra.run.trial130:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:18.008 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.007+0000 7f5cbd948900 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2026-02-18T06:20:18.008 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.007+0000 7f5cbd948900 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2026-02-18T06:20:18.008 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.007+0000 7f5cbd948900 -1 bdev(0x55c933269800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2026-02-18T06:20:18.008 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.008+0000 7f5cbd948900 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2026-02-18T06:20:18.473 DEBUG:teuthology.orchestra.run.trial130:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2026-02-18T06:20:18.539 DEBUG:teuthology.orchestra.run.trial130:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:18.617 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.617+0000 7f8190a25900 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2026-02-18T06:20:18.618 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.617+0000 7f8190a25900 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2026-02-18T06:20:18.618 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.617+0000 7f8190a25900 -1 bdev(0x55bc8e47f800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2026-02-18T06:20:18.618 INFO:teuthology.orchestra.run.trial130.stderr:2026-02-18T06:20:18.617+0000 7f8190a25900 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2026-02-18T06:20:19.096 DEBUG:teuthology.orchestra.run.trial130:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2026-02-18T06:20:19.161 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-18T06:20:19.161 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:20:19.161 DEBUG:teuthology.orchestra.run.trial031:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2026-02-18T06:20:19.186 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.186 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2026-02-18T06:20:19.225 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.225 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2026-02-18T06:20:19.289 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.290 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2026-02-18T06:20:19.353 DEBUG:teuthology.orchestra.run.trial031:> set -ex 2026-02-18T06:20:19.353 DEBUG:teuthology.orchestra.run.trial031:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2026-02-18T06:20:19.371 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.372 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2026-02-18T06:20:19.409 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.410 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2026-02-18T06:20:19.465 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.465 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2026-02-18T06:20:19.522 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.522 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2026-02-18T06:20:19.577 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.577 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2026-02-18T06:20:19.633 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.634 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2026-02-18T06:20:19.691 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.691 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2026-02-18T06:20:19.748 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:19.748 DEBUG:teuthology.orchestra.run.trial130:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2026-02-18T06:20:19.803 INFO:tasks.ceph:Adding keys to all mons... 2026-02-18T06:20:19.804 DEBUG:teuthology.orchestra.run.trial130:> sudo tee -a /etc/ceph/cluster2.keyring 2026-02-18T06:20:19.866 INFO:teuthology.orchestra.run.trial130.stdout:[mgr.x] 2026-02-18T06:20:19.866 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVp7oWlBhAAngyizPrwBUa+Z6uFoOs/qQ== 2026-02-18T06:20:19.866 INFO:teuthology.orchestra.run.trial130.stdout:[osd.0] 2026-02-18T06:20:19.866 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAhWpVpnyEBFxAAuqb/D3XXAdI9Xogzsirzbg== 2026-02-18T06:20:19.866 INFO:teuthology.orchestra.run.trial130.stdout:[osd.1] 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAiWpVpM8B/ABAAHuUJO45KMNgabReyqkObQw== 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout:[osd.2] 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAiWpVp5UnWJBAAU2nfIrQJyhSHQGmfNSLbag== 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout:[client.0] 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpIvTYCBAA0xnJPbd1KfBTTMiKe+aRUg== 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror] 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpPOu8CxAAF36fhedds5Q5t6UDoS8OSQ== 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.0] 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpDNnXEBAAbkFGNwaXAs07mzvyEXg+TA== 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.1] 2026-02-18T06:20:19.867 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpbLYAFhAASz46d0ktZ3MzYIiSTJLD0w== 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.2] 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpk44HGxAA+8XhzT0VuVrA0JaHd6ty4g== 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.3] 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpPPcpIBAA23Yd609EaWRZ+JsbraZw2Q== 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.4] 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpQjg0JRAAGU1zdi7NFwUJ2bsoS8uxUQ== 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.5] 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVp5thQKhAArLwGLIj6eElZW1RywdQdPA== 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.6] 2026-02-18T06:20:19.868 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpZoJwLxAAGDu7iDNSdr1o0j7lISpCQw== 2026-02-18T06:20:19.869 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-18T06:20:19.949 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-18T06:20:20.027 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-18T06:20:20.108 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-18T06:20:20.187 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.268 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.350 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.431 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.511 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.593 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.675 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.756 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.836 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-18T06:20:20.916 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-18T06:20:20.917 DEBUG:teuthology.orchestra.run.trial130:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2026-02-18T06:20:20.981 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2026-02-18T06:20:21.068 DEBUG:teuthology.orchestra.run.trial130:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2026-02-18T06:20:21.096 DEBUG:teuthology.orchestra.run.trial130:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2026-02-18T06:20:21.151 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2026-02-18T06:20:21.152 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2026-02-18T06:20:21.152 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2026-02-18T06:20:21.193 INFO:tasks.ceph.cluster2.mon.a:Started 2026-02-18T06:20:21.194 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2026-02-18T06:20:21.194 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2026-02-18T06:20:21.194 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2026-02-18T06:20:21.197 INFO:tasks.ceph.cluster2.mgr.x:Started 2026-02-18T06:20:21.197 DEBUG:tasks.ceph:set 0 configs 2026-02-18T06:20:21.198 DEBUG:teuthology.orchestra.run.trial130:> sudo ceph --cluster cluster2 config dump 2026-02-18T06:20:21.297 INFO:teuthology.orchestra.run.trial130.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-18T06:20:21.305 INFO:tasks.ceph:Setting crush tunables to default 2026-02-18T06:20:21.306 DEBUG:teuthology.orchestra.run.trial130:> sudo ceph --cluster cluster2 osd crush tunables default 2026-02-18T06:20:21.406 INFO:teuthology.orchestra.run.trial130.stderr:adjusted tunables profile to default 2026-02-18T06:20:21.413 INFO:tasks.ceph:check_enable_crimson: False 2026-02-18T06:20:21.413 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2026-02-18T06:20:21.414 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:21.414 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2026-02-18T06:20:21.438 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:21.439 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2026-02-18T06:20:21.502 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:20:21.502 DEBUG:teuthology.orchestra.run.trial130:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2026-02-18T06:20:21.565 DEBUG:teuthology.orchestra.run.trial130:> sudo ceph --cluster cluster2 osd new 1ae0369e-50e2-440b-8d51-bce7a9a0c0a3 0 2026-02-18T06:20:21.708 INFO:teuthology.orchestra.run.trial130.stdout:0 2026-02-18T06:20:21.716 DEBUG:teuthology.orchestra.run.trial130:> sudo ceph --cluster cluster2 osd new 43da5d80-4629-409f-8241-c6f9b94c3e04 1 2026-02-18T06:20:21.817 INFO:teuthology.orchestra.run.trial130.stdout:1 2026-02-18T06:20:21.825 DEBUG:teuthology.orchestra.run.trial130:> sudo ceph --cluster cluster2 osd new bd78b66b-a532-40e3-a671-f903661484b4 2 2026-02-18T06:20:21.927 INFO:teuthology.orchestra.run.trial130.stdout:2 2026-02-18T06:20:21.935 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2026-02-18T06:20:21.935 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2026-02-18T06:20:21.937 INFO:tasks.ceph.cluster2.osd.0:Started 2026-02-18T06:20:21.937 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2026-02-18T06:20:21.938 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2026-02-18T06:20:21.940 INFO:tasks.ceph.cluster2.osd.1:Started 2026-02-18T06:20:21.940 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2026-02-18T06:20:21.940 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2026-02-18T06:20:21.942 INFO:tasks.ceph.cluster2.osd.2:Started 2026-02-18T06:20:21.943 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-18T06:20:21.953 INFO:tasks.ceph.cluster2.mgr.x.trial031.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-18T06:20:21.953 INFO:tasks.ceph.cluster2.mgr.x.trial031.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-18T06:20:21.953 INFO:tasks.ceph.cluster2.mgr.x.trial031.stderr: from numpy import show_config as show_numpy_config 2026-02-18T06:20:22.043 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:22.044 INFO:teuthology.orchestra.run.trial130.stdout:{"epoch":5,"fsid":"59a365c0-617d-43e5-831e-67d120cf5f4b","created":"2026-02-18T06:20:21.254675+0000","modified":"2026-02-18T06:20:21.927709+0000","last_up_change":"0.000000","last_in_change":"2026-02-18T06:20:21.927709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ae0369e-50e2-440b-8d51-bce7a9a0c0a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"43da5d80-4629-409f-8241-c6f9b94c3e04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bd78b66b-a532-40e3-a671-f903661484b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:22.051 INFO:tasks.ceph.ceph_manager.cluster2:[] 2026-02-18T06:20:22.052 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-18T06:20:22.178 INFO:tasks.ceph.cluster2.osd.0.trial130.stderr:2026-02-18T06:20:22.178+0000 7f68e0810900 -1 Falling back to public interface 2026-02-18T06:20:22.197 INFO:tasks.ceph.cluster2.osd.1.trial130.stderr:2026-02-18T06:20:22.197+0000 7f67ff21f900 -1 Falling back to public interface 2026-02-18T06:20:22.201 INFO:tasks.ceph.cluster2.osd.2.trial130.stderr:2026-02-18T06:20:22.201+0000 7fc4f0e97900 -1 Falling back to public interface 2026-02-18T06:20:22.354 DEBUG:teuthology.orchestra.run.trial130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-18T06:20:22.617 INFO:teuthology.misc.health.trial130.stderr:2026-02-18T06:20:22.616+0000 7fb5feffd640 0 --2- 10.20.193.130:0/1774555846 >> [v2:10.20.193.130:3300/0,v1:10.20.193.130:6789/0] conn(0x7fb600093530 0x7fb60014fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-18T06:20:22.647 INFO:teuthology.misc.health.trial130.stdout: 2026-02-18T06:20:22.648 INFO:teuthology.misc.health.trial130.stdout:{"epoch":5,"fsid":"59a365c0-617d-43e5-831e-67d120cf5f4b","created":"2026-02-18T06:20:21.254675+0000","modified":"2026-02-18T06:20:21.927709+0000","last_up_change":"0.000000","last_in_change":"2026-02-18T06:20:21.927709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ae0369e-50e2-440b-8d51-bce7a9a0c0a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"43da5d80-4629-409f-8241-c6f9b94c3e04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bd78b66b-a532-40e3-a671-f903661484b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:22.655 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-18T06:20:22.705 INFO:tasks.ceph.cluster2.osd.0.trial130.stderr:2026-02-18T06:20:22.705+0000 7f68e0810900 -1 osd.0 0 log_to_monitors true 2026-02-18T06:20:22.722 INFO:tasks.ceph.cluster2.osd.1.trial130.stderr:2026-02-18T06:20:22.722+0000 7f67ff21f900 -1 osd.1 0 log_to_monitors true 2026-02-18T06:20:22.748 INFO:tasks.ceph.cluster2.osd.2.trial130.stderr:2026-02-18T06:20:22.748+0000 7fc4f0e97900 -1 osd.2 0 log_to_monitors true 2026-02-18T06:20:24.144 INFO:tasks.ceph.cluster2.mgr.x.trial031.stderr:2026-02-18T06:20:24.141+0000 7fed7ec71640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-18T06:20:24.276 INFO:tasks.ceph.cluster2.osd.2.trial130.stderr:2026-02-18T06:20:24.276+0000 7fc4ec604640 -1 osd.2 0 waiting for initial osdmap 2026-02-18T06:20:24.276 INFO:tasks.ceph.cluster2.osd.1.trial130.stderr:2026-02-18T06:20:24.276+0000 7f67fa1a7640 -1 osd.1 0 waiting for initial osdmap 2026-02-18T06:20:24.278 INFO:tasks.ceph.cluster2.osd.1.trial130.stderr:2026-02-18T06:20:24.277+0000 7f67f1fb7640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-18T06:20:24.278 INFO:tasks.ceph.cluster2.osd.2.trial130.stderr:2026-02-18T06:20:24.278+0000 7fc4e4414640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-18T06:20:24.279 INFO:tasks.ceph.cluster2.osd.0.trial130.stderr:2026-02-18T06:20:24.279+0000 7f68db678640 -1 osd.0 0 waiting for initial osdmap 2026-02-18T06:20:24.280 INFO:tasks.ceph.cluster2.osd.0.trial130.stderr:2026-02-18T06:20:24.280+0000 7f68d3488640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-18T06:20:28.958 DEBUG:teuthology.orchestra.run.trial130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-18T06:20:29.134 INFO:teuthology.misc.health.trial130.stdout: 2026-02-18T06:20:29.134 INFO:teuthology.misc.health.trial130.stdout:{"epoch":10,"fsid":"59a365c0-617d-43e5-831e-67d120cf5f4b","created":"2026-02-18T06:20:21.254675+0000","modified":"2026-02-18T06:20:28.266991+0000","last_up_change":"2026-02-18T06:20:25.261461+0000","last_in_change":"2026-02-18T06:20:21.927709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:27.157632+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ae0369e-50e2-440b-8d51-bce7a9a0c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6800","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6801","nonce":1721000025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6802","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6803","nonce":1721000025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6806","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6807","nonce":1721000025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6804","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6805","nonce":1721000025}]},"public_addr":"10.20.193.130:6801/1721000025","cluster_addr":"10.20.193.130:6803/1721000025","heartbeat_back_addr":"10.20.193.130:6807/1721000025","heartbeat_front_addr":"10.20.193.130:6805/1721000025","state":["exists","up"]},{"osd":1,"uuid":"43da5d80-4629-409f-8241-c6f9b94c3e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6808","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6809","nonce":370005737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6810","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6811","nonce":370005737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6814","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6815","nonce":370005737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6812","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6813","nonce":370005737}]},"public_addr":"10.20.193.130:6809/370005737","cluster_addr":"10.20.193.130:6811/370005737","heartbeat_back_addr":"10.20.193.130:6815/370005737","heartbeat_front_addr":"10.20.193.130:6813/370005737","state":["exists","up"]},{"osd":2,"uuid":"bd78b66b-a532-40e3-a671-f903661484b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6816","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6817","nonce":1080821858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6818","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6819","nonce":1080821858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6822","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6823","nonce":1080821858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6820","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6821","nonce":1080821858}]},"public_addr":"10.20.193.130:6817/1080821858","cluster_addr":"10.20.193.130:6819/1080821858","heartbeat_back_addr":"10.20.193.130:6823/1080821858","heartbeat_front_addr":"10.20.193.130:6821/1080821858","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:29.142 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-18T06:20:29.143 INFO:tasks.ceph:Creating RBD pool 2026-02-18T06:20:29.143 DEBUG:teuthology.orchestra.run.trial130:> sudo ceph --cluster cluster2 osd pool create rbd 8 2026-02-18T06:20:30.271 INFO:teuthology.orchestra.run.trial130.stderr:pool 'rbd' created 2026-02-18T06:20:30.285 DEBUG:teuthology.orchestra.run.trial130:> rbd --cluster cluster2 pool init rbd 2026-02-18T06:20:33.282 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2026-02-18T06:20:33.283 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 config log 1 --format=json 2026-02-18T06:20:33.284 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-18T06:20:33.466 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:33.475 INFO:teuthology.orchestra.run.trial130.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-18T06:20:33.475 INFO:tasks.ceph_manager:config epoch is 1 2026-02-18T06:20:33.476 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2026-02-18T06:20:33.476 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2026-02-18T06:20:33.476 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2026-02-18T06:20:33.687 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:33.695 INFO:teuthology.orchestra.run.trial130.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6826","nonce":700274964},{"type":"v1","addr":"10.20.193.31:6827","nonce":700274964}]},"active_addr":"10.20.193.31:6827/700274964","active_change":"2026-02-18T06:20:23.139096+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":1621686115}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":1499631570}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":2307962683}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.31:0","nonce":1582105479}]}]} 2026-02-18T06:20:33.698 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2026-02-18T06:20:33.699 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2026-02-18T06:20:33.699 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-18T06:20:33.881 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:33.882 INFO:teuthology.orchestra.run.trial130.stdout:{"epoch":15,"fsid":"59a365c0-617d-43e5-831e-67d120cf5f4b","created":"2026-02-18T06:20:21.254675+0000","modified":"2026-02-18T06:20:33.276577+0000","last_up_change":"2026-02-18T06:20:25.261461+0000","last_in_change":"2026-02-18T06:20:21.927709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:27.157632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-18T06:20:29.327794+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ae0369e-50e2-440b-8d51-bce7a9a0c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6800","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6801","nonce":1721000025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6802","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6803","nonce":1721000025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6806","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6807","nonce":1721000025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6804","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6805","nonce":1721000025}]},"public_addr":"10.20.193.130:6801/1721000025","cluster_addr":"10.20.193.130:6803/1721000025","heartbeat_back_addr":"10.20.193.130:6807/1721000025","heartbeat_front_addr":"10.20.193.130:6805/1721000025","state":["exists","up"]},{"osd":1,"uuid":"43da5d80-4629-409f-8241-c6f9b94c3e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6808","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6809","nonce":370005737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6810","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6811","nonce":370005737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6814","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6815","nonce":370005737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6812","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6813","nonce":370005737}]},"public_addr":"10.20.193.130:6809/370005737","cluster_addr":"10.20.193.130:6811/370005737","heartbeat_back_addr":"10.20.193.130:6815/370005737","heartbeat_front_addr":"10.20.193.130:6813/370005737","state":["exists","up"]},{"osd":2,"uuid":"bd78b66b-a532-40e3-a671-f903661484b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6816","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6817","nonce":1080821858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6818","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6819","nonce":1080821858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6822","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6823","nonce":1080821858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6820","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6821","nonce":1080821858}]},"public_addr":"10.20.193.130:6817/1080821858","cluster_addr":"10.20.193.130:6819/1080821858","heartbeat_back_addr":"10.20.193.130:6823/1080821858","heartbeat_front_addr":"10.20.193.130:6821/1080821858","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:33.892 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2026-02-18T06:20:33.892 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-18T06:20:34.074 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:34.074 INFO:teuthology.orchestra.run.trial130.stdout:{"epoch":15,"fsid":"59a365c0-617d-43e5-831e-67d120cf5f4b","created":"2026-02-18T06:20:21.254675+0000","modified":"2026-02-18T06:20:33.276577+0000","last_up_change":"2026-02-18T06:20:25.261461+0000","last_in_change":"2026-02-18T06:20:21.927709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:27.157632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-18T06:20:29.327794+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ae0369e-50e2-440b-8d51-bce7a9a0c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6800","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6801","nonce":1721000025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6802","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6803","nonce":1721000025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6806","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6807","nonce":1721000025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6804","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6805","nonce":1721000025}]},"public_addr":"10.20.193.130:6801/1721000025","cluster_addr":"10.20.193.130:6803/1721000025","heartbeat_back_addr":"10.20.193.130:6807/1721000025","heartbeat_front_addr":"10.20.193.130:6805/1721000025","state":["exists","up"]},{"osd":1,"uuid":"43da5d80-4629-409f-8241-c6f9b94c3e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6808","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6809","nonce":370005737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6810","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6811","nonce":370005737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6814","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6815","nonce":370005737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6812","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6813","nonce":370005737}]},"public_addr":"10.20.193.130:6809/370005737","cluster_addr":"10.20.193.130:6811/370005737","heartbeat_back_addr":"10.20.193.130:6815/370005737","heartbeat_front_addr":"10.20.193.130:6813/370005737","state":["exists","up"]},{"osd":2,"uuid":"bd78b66b-a532-40e3-a671-f903661484b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6816","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6817","nonce":1080821858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6818","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6819","nonce":1080821858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6822","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6823","nonce":1080821858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6820","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6821","nonce":1080821858}]},"public_addr":"10.20.193.130:6817/1080821858","cluster_addr":"10.20.193.130:6819/1080821858","heartbeat_back_addr":"10.20.193.130:6823/1080821858","heartbeat_front_addr":"10.20.193.130:6821/1080821858","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:20:34.084 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2026-02-18T06:20:34.085 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2026-02-18T06:20:34.085 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2026-02-18T06:20:34.174 INFO:teuthology.orchestra.run.trial130.stdout:34359738372 2026-02-18T06:20:34.175 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-18T06:20:34.176 INFO:teuthology.orchestra.run.trial130.stdout:34359738373 2026-02-18T06:20:34.176 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-18T06:20:34.177 INFO:teuthology.orchestra.run.trial130.stdout:34359738372 2026-02-18T06:20:34.178 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-18T06:20:34.399 INFO:teuthology.orchestra.run.trial130.stdout:34359738371 2026-02-18T06:20:34.405 INFO:teuthology.orchestra.run.trial130.stdout:34359738372 2026-02-18T06:20:34.407 INFO:teuthology.orchestra.run.trial130.stdout:34359738371 2026-02-18T06:20:34.409 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2026-02-18T06:20:34.416 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2026-02-18T06:20:34.418 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.1 2026-02-18T06:20:35.410 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-18T06:20:35.416 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-18T06:20:35.419 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-18T06:20:35.593 INFO:teuthology.orchestra.run.trial130.stdout:34359738372 2026-02-18T06:20:35.604 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2026-02-18T06:20:35.604 DEBUG:teuthology.parallel:result is None 2026-02-18T06:20:35.639 INFO:teuthology.orchestra.run.trial130.stdout:34359738372 2026-02-18T06:20:35.640 INFO:teuthology.orchestra.run.trial130.stdout:34359738373 2026-02-18T06:20:35.649 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.1 2026-02-18T06:20:35.649 DEBUG:teuthology.parallel:result is None 2026-02-18T06:20:35.650 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2026-02-18T06:20:35.650 DEBUG:teuthology.parallel:result is None 2026-02-18T06:20:35.650 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-18T06:20:35.650 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-18T06:20:35.870 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:35.870 INFO:teuthology.orchestra.run.trial130.stderr:dumped all 2026-02-18T06:20:35.879 INFO:teuthology.orchestra.run.trial130.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-18T06:20:35.142527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82416,"kb_used_data":1536,"kb_used_omap":20,"kb_used_meta":80811,"kb_avail":283033104,"statfs":{"total":289910292480,"available":289825898496,"internally_reserved":0,"allocated":1572864,"data_stored":1288327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21124,"internal_metadata":82750844},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.872822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279922+0000","last_change":"2026-02-18T06:20:33.279922+0000","last_active":"2026-02-18T06:20:33.279922+0000","last_peered":"2026-02-18T06:20:33.279922+0000","last_clean":"2026-02-18T06:20:33.279922+0000","last_became_active":"2026-02-18T06:20:31.279056+0000","last_became_peered":"2026-02-18T06:20:31.279056+0000","last_unstale":"2026-02-18T06:20:33.279922+0000","last_undegraded":"2026-02-18T06:20:33.279922+0000","last_fullsized":"2026-02-18T06:20:33.279922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:34:10.245401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053659699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279639+0000","last_change":"2026-02-18T06:20:33.279639+0000","last_active":"2026-02-18T06:20:33.279639+0000","last_peered":"2026-02-18T06:20:33.279639+0000","last_clean":"2026-02-18T06:20:33.279639+0000","last_became_active":"2026-02-18T06:20:31.279113+0000","last_became_peered":"2026-02-18T06:20:31.279113+0000","last_unstale":"2026-02-18T06:20:33.279639+0000","last_undegraded":"2026-02-18T06:20:33.279639+0000","last_fullsized":"2026-02-18T06:20:33.279639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:36:05.161609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030664000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279798+0000","last_change":"2026-02-18T06:20:33.279798+0000","last_active":"2026-02-18T06:20:33.279798+0000","last_peered":"2026-02-18T06:20:33.279798+0000","last_clean":"2026-02-18T06:20:33.279798+0000","last_became_active":"2026-02-18T06:20:31.279035+0000","last_became_peered":"2026-02-18T06:20:31.279035+0000","last_unstale":"2026-02-18T06:20:33.279798+0000","last_undegraded":"2026-02-18T06:20:33.279798+0000","last_fullsized":"2026-02-18T06:20:33.279798+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:28:19.490146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043416700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279641+0000","last_change":"2026-02-18T06:20:33.279641+0000","last_active":"2026-02-18T06:20:33.279641+0000","last_peered":"2026-02-18T06:20:33.279641+0000","last_clean":"2026-02-18T06:20:33.279641+0000","last_became_active":"2026-02-18T06:20:31.278965+0000","last_became_peered":"2026-02-18T06:20:31.278965+0000","last_unstale":"2026-02-18T06:20:33.279641+0000","last_undegraded":"2026-02-18T06:20:33.279641+0000","last_fullsized":"2026-02-18T06:20:33.279641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:25:34.395735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027168500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279089+0000","last_change":"2026-02-18T06:20:33.278364+0000","last_active":"2026-02-18T06:20:33.279089+0000","last_peered":"2026-02-18T06:20:33.279089+0000","last_clean":"2026-02-18T06:20:33.279089+0000","last_became_active":"2026-02-18T06:20:31.279064+0000","last_became_peered":"2026-02-18T06:20:31.279064+0000","last_unstale":"2026-02-18T06:20:33.279089+0000","last_undegraded":"2026-02-18T06:20:33.279089+0000","last_fullsized":"2026-02-18T06:20:33.279089+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:11:20.458617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020610299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.748278+0000","last_change":"2026-02-18T06:20:33.748366+0000","last_active":"2026-02-18T06:20:33.748278+0000","last_peered":"2026-02-18T06:20:33.748278+0000","last_clean":"2026-02-18T06:20:33.748278+0000","last_became_active":"2026-02-18T06:20:31.279036+0000","last_became_peered":"2026-02-18T06:20:31.279036+0000","last_unstale":"2026-02-18T06:20:33.748278+0000","last_undegraded":"2026-02-18T06:20:33.748278+0000","last_fullsized":"2026-02-18T06:20:33.748278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:52:51.049171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017900300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.748258+0000","last_change":"2026-02-18T06:20:33.748330+0000","last_active":"2026-02-18T06:20:33.748258+0000","last_peered":"2026-02-18T06:20:33.748258+0000","last_clean":"2026-02-18T06:20:33.748258+0000","last_became_active":"2026-02-18T06:20:31.279067+0000","last_became_peered":"2026-02-18T06:20:31.279067+0000","last_unstale":"2026-02-18T06:20:33.748258+0000","last_undegraded":"2026-02-18T06:20:33.748258+0000","last_fullsized":"2026-02-18T06:20:33.748258+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:55:16.535994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017421299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279767+0000","last_change":"2026-02-18T06:20:33.279767+0000","last_active":"2026-02-18T06:20:33.279767+0000","last_peered":"2026-02-18T06:20:33.279767+0000","last_clean":"2026-02-18T06:20:33.279767+0000","last_became_active":"2026-02-18T06:20:31.278647+0000","last_became_peered":"2026-02-18T06:20:31.278647+0000","last_unstale":"2026-02-18T06:20:33.279767+0000","last_undegraded":"2026-02-18T06:20:33.279767+0000","last_fullsized":"2026-02-18T06:20:33.279767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:33:35.594020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040074499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":111,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279212+0000","last_change":"2026-02-18T06:20:28.711581+0000","last_active":"2026-02-18T06:20:33.279212+0000","last_peered":"2026-02-18T06:20:33.279212+0000","last_clean":"2026-02-18T06:20:33.279212+0000","last_became_active":"2026-02-18T06:20:28.710726+0000","last_became_peered":"2026-02-18T06:20:28.710726+0000","last_unstale":"2026-02-18T06:20:33.279212+0000","last_undegraded":"2026-02-18T06:20:33.279212+0000","last_fullsized":"2026-02-18T06:20:33.279212+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_clean_scrub_stamp":"2026-02-18T06:20:27.265730+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T09:48:59.040538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27072,"kb_used_data":112,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344768,"statfs":{"total":96636764160,"available":96609042432,"internally_reserved":0,"allocated":114688,"data_stored":31155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6825,"internal_metadata":27583831},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8774,"internal_metadata":27581882},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5525,"internal_metadata":27585131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:20:35.880 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-18T06:20:36.059 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:36.059 INFO:teuthology.orchestra.run.trial130.stderr:dumped all 2026-02-18T06:20:36.068 INFO:teuthology.orchestra.run.trial130.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-18T06:20:35.142527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82416,"kb_used_data":1536,"kb_used_omap":20,"kb_used_meta":80811,"kb_avail":283033104,"statfs":{"total":289910292480,"available":289825898496,"internally_reserved":0,"allocated":1572864,"data_stored":1288327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21124,"internal_metadata":82750844},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.872822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279922+0000","last_change":"2026-02-18T06:20:33.279922+0000","last_active":"2026-02-18T06:20:33.279922+0000","last_peered":"2026-02-18T06:20:33.279922+0000","last_clean":"2026-02-18T06:20:33.279922+0000","last_became_active":"2026-02-18T06:20:31.279056+0000","last_became_peered":"2026-02-18T06:20:31.279056+0000","last_unstale":"2026-02-18T06:20:33.279922+0000","last_undegraded":"2026-02-18T06:20:33.279922+0000","last_fullsized":"2026-02-18T06:20:33.279922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:34:10.245401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053659699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279639+0000","last_change":"2026-02-18T06:20:33.279639+0000","last_active":"2026-02-18T06:20:33.279639+0000","last_peered":"2026-02-18T06:20:33.279639+0000","last_clean":"2026-02-18T06:20:33.279639+0000","last_became_active":"2026-02-18T06:20:31.279113+0000","last_became_peered":"2026-02-18T06:20:31.279113+0000","last_unstale":"2026-02-18T06:20:33.279639+0000","last_undegraded":"2026-02-18T06:20:33.279639+0000","last_fullsized":"2026-02-18T06:20:33.279639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:36:05.161609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030664000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279798+0000","last_change":"2026-02-18T06:20:33.279798+0000","last_active":"2026-02-18T06:20:33.279798+0000","last_peered":"2026-02-18T06:20:33.279798+0000","last_clean":"2026-02-18T06:20:33.279798+0000","last_became_active":"2026-02-18T06:20:31.279035+0000","last_became_peered":"2026-02-18T06:20:31.279035+0000","last_unstale":"2026-02-18T06:20:33.279798+0000","last_undegraded":"2026-02-18T06:20:33.279798+0000","last_fullsized":"2026-02-18T06:20:33.279798+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:28:19.490146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043416700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279641+0000","last_change":"2026-02-18T06:20:33.279641+0000","last_active":"2026-02-18T06:20:33.279641+0000","last_peered":"2026-02-18T06:20:33.279641+0000","last_clean":"2026-02-18T06:20:33.279641+0000","last_became_active":"2026-02-18T06:20:31.278965+0000","last_became_peered":"2026-02-18T06:20:31.278965+0000","last_unstale":"2026-02-18T06:20:33.279641+0000","last_undegraded":"2026-02-18T06:20:33.279641+0000","last_fullsized":"2026-02-18T06:20:33.279641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:25:34.395735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027168500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279089+0000","last_change":"2026-02-18T06:20:33.278364+0000","last_active":"2026-02-18T06:20:33.279089+0000","last_peered":"2026-02-18T06:20:33.279089+0000","last_clean":"2026-02-18T06:20:33.279089+0000","last_became_active":"2026-02-18T06:20:31.279064+0000","last_became_peered":"2026-02-18T06:20:31.279064+0000","last_unstale":"2026-02-18T06:20:33.279089+0000","last_undegraded":"2026-02-18T06:20:33.279089+0000","last_fullsized":"2026-02-18T06:20:33.279089+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:11:20.458617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020610299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.748278+0000","last_change":"2026-02-18T06:20:33.748366+0000","last_active":"2026-02-18T06:20:33.748278+0000","last_peered":"2026-02-18T06:20:33.748278+0000","last_clean":"2026-02-18T06:20:33.748278+0000","last_became_active":"2026-02-18T06:20:31.279036+0000","last_became_peered":"2026-02-18T06:20:31.279036+0000","last_unstale":"2026-02-18T06:20:33.748278+0000","last_undegraded":"2026-02-18T06:20:33.748278+0000","last_fullsized":"2026-02-18T06:20:33.748278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:52:51.049171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017900300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.748258+0000","last_change":"2026-02-18T06:20:33.748330+0000","last_active":"2026-02-18T06:20:33.748258+0000","last_peered":"2026-02-18T06:20:33.748258+0000","last_clean":"2026-02-18T06:20:33.748258+0000","last_became_active":"2026-02-18T06:20:31.279067+0000","last_became_peered":"2026-02-18T06:20:31.279067+0000","last_unstale":"2026-02-18T06:20:33.748258+0000","last_undegraded":"2026-02-18T06:20:33.748258+0000","last_fullsized":"2026-02-18T06:20:33.748258+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:55:16.535994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017421299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279767+0000","last_change":"2026-02-18T06:20:33.279767+0000","last_active":"2026-02-18T06:20:33.279767+0000","last_peered":"2026-02-18T06:20:33.279767+0000","last_clean":"2026-02-18T06:20:33.279767+0000","last_became_active":"2026-02-18T06:20:31.278647+0000","last_became_peered":"2026-02-18T06:20:31.278647+0000","last_unstale":"2026-02-18T06:20:33.279767+0000","last_undegraded":"2026-02-18T06:20:33.279767+0000","last_fullsized":"2026-02-18T06:20:33.279767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:33:35.594020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040074499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":111,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-18T06:20:33.279212+0000","last_change":"2026-02-18T06:20:28.711581+0000","last_active":"2026-02-18T06:20:33.279212+0000","last_peered":"2026-02-18T06:20:33.279212+0000","last_clean":"2026-02-18T06:20:33.279212+0000","last_became_active":"2026-02-18T06:20:28.710726+0000","last_became_peered":"2026-02-18T06:20:28.710726+0000","last_unstale":"2026-02-18T06:20:33.279212+0000","last_undegraded":"2026-02-18T06:20:33.279212+0000","last_fullsized":"2026-02-18T06:20:33.279212+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_clean_scrub_stamp":"2026-02-18T06:20:27.265730+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T09:48:59.040538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27072,"kb_used_data":112,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":94344768,"statfs":{"total":96636764160,"available":96609042432,"internally_reserved":0,"allocated":114688,"data_stored":31155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6825,"internal_metadata":27583831},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8774,"internal_metadata":27581882},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":712,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":729088,"data_stored":628586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5525,"internal_metadata":27585131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:20:36.069 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-18T06:20:36.069 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2026-02-18T06:20:36.069 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2026-02-18T06:20:36.069 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2026-02-18T06:20:36.275 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:20:36.275 INFO:teuthology.orchestra.run.trial130.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-18T06:20:36.285 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2026-02-18T06:20:36.286 INFO:teuthology.run_tasks:Running task exec... 2026-02-18T06:20:36.299 INFO:teuthology.task.exec:Executing custom commands... 2026-02-18T06:20:36.299 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:20:36.299 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:36.531 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.4] 2026-02-18T06:20:36.531 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVpeSxDHBAAuQsO/vgyPZB9w71SRGuePQ== 2026-02-18T06:20:36.531 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:36.531 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:36.531 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.4 2026-02-18T06:20:36.540 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:36.770 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.5] 2026-02-18T06:20:36.771 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVpDkJGIRAAkjMFj/1dIV7rNKFFtPg3gg== 2026-02-18T06:20:36.771 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:36.771 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:36.771 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.5 2026-02-18T06:20:36.781 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:37.010 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.6] 2026-02-18T06:20:37.010 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVpf3JPJhAAzUnvPuAVHlMxl7KuSltkIA== 2026-02-18T06:20:37.010 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:37.010 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:37.011 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.6 2026-02-18T06:20:37.020 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:37.251 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.4] 2026-02-18T06:20:37.252 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpQjg0JRAAGU1zdi7NFwUJ2bsoS8uxUQ== 2026-02-18T06:20:37.252 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:37.252 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:37.252 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.4 2026-02-18T06:20:37.262 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:37.495 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.5] 2026-02-18T06:20:37.496 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVp5thQKhAArLwGLIj6eElZW1RywdQdPA== 2026-02-18T06:20:37.496 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:37.496 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:37.496 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.5 2026-02-18T06:20:37.506 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:37.738 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.6] 2026-02-18T06:20:37.738 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpZoJwLxAAGDu7iDNSdr1o0j7lISpCQw== 2026-02-18T06:20:37.738 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:37.738 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:37.739 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.6 2026-02-18T06:20:37.749 INFO:teuthology.run_tasks:Running task exec... 2026-02-18T06:20:37.764 INFO:teuthology.task.exec:Executing custom commands... 2026-02-18T06:20:37.764 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:20:37.764 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:37.998 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror] 2026-02-18T06:20:37.998 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVp1n0VAxAAmfwRY6GefheReh7l824hLQ== 2026-02-18T06:20:37.998 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-18T06:20:37.999 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:37.999 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror 2026-02-18T06:20:38.009 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:38.240 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.0] 2026-02-18T06:20:38.240 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVpUeAyCBAAK7O9/L3Caf/+xGXNz1wlFQ== 2026-02-18T06:20:38.240 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:38.241 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:38.241 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.0 2026-02-18T06:20:38.250 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:38.479 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.1] 2026-02-18T06:20:38.479 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVpFKZHDRAAMbnGo7HW52CSq5VLpIVi7Q== 2026-02-18T06:20:38.480 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:38.480 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:38.480 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.1 2026-02-18T06:20:38.489 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:38.719 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.2] 2026-02-18T06:20:38.719 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVpisZDEhAAmYxw6HD0NrD/RkCKJOON2A== 2026-02-18T06:20:38.719 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:38.719 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:38.719 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.2 2026-02-18T06:20:38.729 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:38.958 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.3] 2026-02-18T06:20:38.958 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAIWpVpvTxOFxAAb06so6zFGY8xZMII2wqzOg== 2026-02-18T06:20:38.958 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:38.958 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:38.958 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.3 2026-02-18T06:20:38.968 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:39.200 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror] 2026-02-18T06:20:39.200 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpPOu8CxAAF36fhedds5Q5t6UDoS8OSQ== 2026-02-18T06:20:39.200 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-18T06:20:39.200 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:39.200 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror 2026-02-18T06:20:39.211 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:39.441 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.0] 2026-02-18T06:20:39.442 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpDNnXEBAAbkFGNwaXAs07mzvyEXg+TA== 2026-02-18T06:20:39.442 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:39.442 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:39.442 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.0 2026-02-18T06:20:39.452 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:39.685 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.1] 2026-02-18T06:20:39.685 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpbLYAFhAASz46d0ktZ3MzYIiSTJLD0w== 2026-02-18T06:20:39.685 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:39.685 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:39.685 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.1 2026-02-18T06:20:39.695 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:39.926 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.2] 2026-02-18T06:20:39.926 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpk44HGxAA+8XhzT0VuVrA0JaHd6ty4g== 2026-02-18T06:20:39.926 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:39.926 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:39.926 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.2 2026-02-18T06:20:39.937 DEBUG:teuthology.orchestra.run.trial130:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-18T06:20:40.165 INFO:teuthology.orchestra.run.trial130.stdout:[client.mirror.3] 2026-02-18T06:20:40.166 INFO:teuthology.orchestra.run.trial130.stdout: key = AQAeWpVpPPcpIBAA23Yd609EaWRZ+JsbraZw2Q== 2026-02-18T06:20:40.166 INFO:teuthology.orchestra.run.trial130.stdout: caps mon = "profile rbd-mirror" 2026-02-18T06:20:40.166 INFO:teuthology.orchestra.run.trial130.stdout: caps osd = "profile rbd" 2026-02-18T06:20:40.166 INFO:teuthology.orchestra.run.trial130.stderr:updated caps for client.mirror.3 2026-02-18T06:20:40.175 INFO:teuthology.run_tasks:Running task workunit... 2026-02-18T06:20:40.190 INFO:tasks.workunit:Pulling workunits from ref adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T06:20:40.190 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-18T06:20:40.190 DEBUG:teuthology.orchestra.run.trial130:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-18T06:20:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:20:40.206 INFO:teuthology.orchestra.run.trial130.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2026-02-18T06:20:40.206 DEBUG:teuthology.orchestra.run.trial130:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-18T06:20:40.262 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-18T06:20:40.262 DEBUG:teuthology.orchestra.run.trial130:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2026-02-18T06:20:40.318 INFO:tasks.workunit:timeout=3h 2026-02-18T06:20:40.318 INFO:tasks.workunit:cleanup=True 2026-02-18T06:20:40.318 DEBUG:teuthology.orchestra.run.trial130:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T06:20:40.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2026-02-18T06:21:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Note: switching to 'adc1923395f2bcd6ae89b5d331d278364d1848ed'. 2026-02-18T06:21:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:state without impacting any branches by switching back to a branch. 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: git switch -c 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Or undo this operation with: 2026-02-18T06:21:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:21:00.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: git switch - 2026-02-18T06:21:00.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:21:00.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-18T06:21:00.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:21:00.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:HEAD is now at adc1923395f Merge PR #67279 into tentacle 2026-02-18T06:21:00.132 DEBUG:teuthology.orchestra.run.trial130:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2026-02-18T06:21:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-18T06:21:00.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-18T06:21:00.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-18T06:21:00.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-18T06:21:00.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-18T06:21:00.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-18T06:21:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-18T06:21:00.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-18T06:21:00.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-18T06:21:00.377 DEBUG:teuthology.orchestra.run.trial130:> set -ex 2026-02-18T06:21:00.377 DEBUG:teuthology.orchestra.run.trial130:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2026-02-18T06:21:00.434 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror.sh on cluster1.client.mirror... 2026-02-18T06:21:00.436 INFO:tasks.workunit:Running workunit rbd/rbd_mirror.sh... 2026-02-18T06:21:00.437 DEBUG:teuthology.orchestra.run.trial130:workunit test rbd/rbd_mirror.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=journal RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh 2026-02-18T06:21:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -gt 0 ']' 2026-02-18T06:21:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal '!=' snapshot ']' 2026-02-18T06:21:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal '!=' journal ']' 2026-02-18T06:21:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ RBD_MIRROR_INSTANCES=4 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CLUSTER1=cluster1 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CLUSTER2=cluster2 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ PEER_CLUSTER_SUFFIX= 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ POOL=mirror 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ PARENT_POOL=mirror_parent 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ NS1=ns1 2026-02-18T06:21:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ NS2=ns2 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ TEMPDIR= 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ID=mirror 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ RBD_MIRROR_MODE=journal 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ MIRROR_POOL_MODE=pool 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' journal = snapshot ']' 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ export 'CEPH_ARGS=--id mirror' 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS='--id mirror' 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ LAST_MIRROR_INSTANCE=3 2026-02-18T06:21:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh 2026-02-18T06:21:00.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2026-02-18T06:21:00.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2026-02-18T06:21:00.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_BIN=. 2026-02-18T06:21:00.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_SRC=. 2026-02-18T06:21:00.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' -e CMakeCache.txt ']' 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 -gt 0 ']' 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ setup 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local c 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ trap 'cleanup $?' INT TERM EXIT 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ setup_tempdir 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mktemp -d 2026-02-18T06:21:00.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ TEMPDIR=/tmp/tmp.mzUeR8FHMy 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z 1 ']' 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ setup_pools cluster1 cluster2 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster2 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mon_map_file 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mon_addr 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local admin_key_file 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2026-02-18T06:21:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror' already exists 2026-02-18T06:21:01.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:01.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2026-02-18T06:21:02.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror_parent' already exists 2026-02-18T06:21:02.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:02.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 pool init mirror 2026-02-18T06:21:05.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:05.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2026-02-18T06:21:08.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2026-02-18T06:21:08.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2026-02-18T06:21:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2026-02-18T06:21:08.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2026-02-18T06:21:08.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 namespace create mirror_parent/ns1 2026-02-18T06:21:08.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2026-02-18T06:21:08.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2026-02-18T06:21:08.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent/ns1 pool 2026-02-18T06:21:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z ']' 2026-02-18T06:21:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z ']' 2026-02-18T06:21:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_add cluster1 mirror cluster2 2026-02-18T06:21:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local client_cluster=cluster2 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster2 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid_var_name 2026-02-18T06:21:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:08.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local error_code 2026-02-18T06:21:08.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local peer_uuid 2026-02-18T06:21:08.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 16 32 2026-02-18T06:21:08.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:21:08.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_uuid=d037f6ff-011b-4052-abe8-144fb9e70771 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 17 ']' 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 -eq 0 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_add cluster1 mirror_parent cluster2 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local client_cluster=cluster2 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster2 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid_var_name 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local error_code 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local peer_uuid 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 16 32 2026-02-18T06:21:08.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:21:08.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_uuid=16035ff9-d0e7-45fb-8544-bc6c842315e4 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 17 ']' 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 -eq 0 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ setup_pools cluster2 cluster1 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:08.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster1 2026-02-18T06:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mon_map_file 2026-02-18T06:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mon_addr 2026-02-18T06:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local admin_key_file 2026-02-18T06:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid 2026-02-18T06:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:08.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2026-02-18T06:21:09.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror' already exists 2026-02-18T06:21:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2026-02-18T06:21:10.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror_parent' already exists 2026-02-18T06:21:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:10.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 pool init mirror 2026-02-18T06:21:13.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:13.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2026-02-18T06:21:16.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:16.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:21:16.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2026-02-18T06:21:16.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2026-02-18T06:21:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2026-02-18T06:21:16.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2026-02-18T06:21:16.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 namespace create mirror_parent/ns1 2026-02-18T06:21:16.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2026-02-18T06:21:16.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2026-02-18T06:21:16.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent/ns1 pool 2026-02-18T06:21:16.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z ']' 2026-02-18T06:21:16.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z ']' 2026-02-18T06:21:16.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_add cluster2 mirror cluster1 2026-02-18T06:21:16.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:16.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local client_cluster=cluster1 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster1 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid_var_name 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local error_code 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local peer_uuid 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 16 32 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:21:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2026-02-18T06:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_uuid=1f432250-efc6-4bf5-b06a-fd508092746d 2026-02-18T06:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 17 ']' 2026-02-18T06:21:16.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 -eq 0 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_add cluster2 mirror_parent cluster1 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local client_cluster=cluster1 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster1 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid_var_name 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:16.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local error_code 2026-02-18T06:21:16.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local peer_uuid 2026-02-18T06:21:16.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 16 32 2026-02-18T06:21:16.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:21:16.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_uuid=d60b2ae1-4124-483b-8615-16f580763e0c 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 17 ']' 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 -eq 0 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:16.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: mirror from default namespace to non-default namespace' 2026-02-18T06:21:16.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:21:16.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:21:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:21:16 TEST: mirror from default namespace to non-default namespace 2026-02-18T06:21:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:21:16 TEST: mirror from default namespace to non-default namespace 2026-02-18T06:21:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster1 2026-02-18T06:21:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:0 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:16.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:16.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:16.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:1 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:16.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:2 2026-02-18T06:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:21:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:3 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:21:16.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:16.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:16.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:16.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:21:16.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:16.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:16.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:16.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image=test 2026-02-18T06:21:16.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror init-only 2026-02-18T06:21:16.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror init-only 2026-02-18T06:21:16.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool disable mirror/ns1 2026-02-18T06:21:16.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool disable mirror/ns1 2026-02-18T06:21:16.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror pool --remote-namespace ns1 2026-02-18T06:21:16.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool --remote-namespace '' 2026-02-18T06:21:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 2026-02-18T06:21:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:21:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2026-02-18T06:21:16.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:21:16.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:16.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test started 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:21:16.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:16.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror/ns1 test started 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:21:17.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:21:17.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:17.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:21:17.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:17.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:17.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:17.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:17.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:17.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:17.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:17.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:17.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:17.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:17.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:17.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:17.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:17.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:17.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:17.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:17.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:17.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:17.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:17.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:17.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:17.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:17.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:17.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:17.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:17.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:17.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:17.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:17.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:17.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:17.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:17.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:21:18.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:18.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:18.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:18.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:18.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:19.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:19.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:19.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:19.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:19.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:19.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:19.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:19.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test 2026-02-18T06:21:19.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:19.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:19.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:19.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:19.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:19.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:19.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:19.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:19.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:19.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:19.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:19.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:19.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:19.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:19.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:19.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:19.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:19.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:19.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:19.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:19.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:19.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:19.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:19.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:21:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:21.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:21.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:21.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:21.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:21.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:21.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:21.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:21.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:21.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:21.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:21.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:21.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:21.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:21.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:21.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:21.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:21.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:21.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:21.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:21.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:21.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:21.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:21.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:21.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:21.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:21.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:21:21.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:21.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:21.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:21.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:21:25.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:25.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:25.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:25.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-18T06:21:25.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:25.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:25.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:25.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:25.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:25.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:25.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:25.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:25.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:25.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:25.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:25.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-18T06:21:25.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:21:25.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:21:25.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:21:25.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:21:25.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test 100 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:21:25.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:21:25.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:25.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:21:25.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:21:25.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:21:25.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:21:25.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.5 bytes/sec: 30 MiB/s 2026-02-18T06:21:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 mirror test 2026-02-18T06:21:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 mirror test 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:21:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:21:25.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:25.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:21:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test 2026-02-18T06:21:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:25.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:25.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:25.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:25.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:25.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:25.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:25.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:25.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:25.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:25.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:25.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:25.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:25.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:25.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:25.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2026-02-18T06:21:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:21:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:25.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:21:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:25.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:25.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:25.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:25.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:25.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:25.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:25.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:25.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:25.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:25.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:25.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:25.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:21:25.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:25.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' '' 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = '' 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:21:25.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:21:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:21:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:21:25.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:21:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:21:25.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:25.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:21:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:21:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:21:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=1 2026-02-18T06:21:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 1 -a 101 -le 1 2026-02-18T06:21:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:21:25.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:25.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:21:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test 2026-02-18T06:21:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:26.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:26.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:26.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2026-02-18T06:21:26.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:26.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:21:26.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:26.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:26.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:26.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=2, entry_tid=32], [object_number=3, tag_tid=2, entry_tid=31], [object_number=2, tag_tid=2, entry_tid=30], [object_number=1, tag_tid=2, entry_tid=29]]], state=connected] 2026-02-18T06:21:26.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:26.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:21:26.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:26.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:26.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:26.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=2, entry_tid=32], [object_number=3, tag_tid=2, entry_tid=31], [object_number=2, tag_tid=2, entry_tid=30], [object_number=1, tag_tid=2, entry_tid=29]]], state=connected] 2026-02-18T06:21:26.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=2, entry_tid=32' 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=0, tag_tid=2, entry_tid=32' 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=2, entry_tid=32' '!=' 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=2, entry_tid=32' = 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=0, tag_tid=2, entry_tid=32' 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:21:26.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:21:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=2, entry_tid=32' 2026-02-18T06:21:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:21:26.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:26.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=2 2026-02-18T06:21:26.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:26.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:21:26.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:26.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:21:26.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=2, entry_tid=32' 2026-02-18T06:21:26.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:21:26.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:26.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=32 2026-02-18T06:21:26.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 2 -a 101 -le 32 2026-02-18T06:21:26.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:21:26.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:26.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test 2026-02-18T06:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:26.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:26.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:26.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:26.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:26.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:26.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:26.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:26.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:26.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:26.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:26.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:26.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:26.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:26.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2026-02-18T06:21:26.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:26.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:26.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:26.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:26.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:26.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:26.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:21:26.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:26.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:26.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:26.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:26.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:26.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:26.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:26.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns1 test 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:26.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:26.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position '' 2026-02-18T06:21:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:27.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:27.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test 2026-02-18T06:21:27.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:21:27.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 089f4760-8ec0-4891-8cf5-df62c4bbf216 2026-02-18T06:21:27.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:21:27.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:21:25' 2026-02-18T06:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:21:27.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:21:27.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:21:27.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:21:27.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:21:27.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:21:27.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:21:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:21:25' 2026-02-18T06:21:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test up+replaying '2026-02-18 06:21:25' 2026-02-18T06:21:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:27.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:21:25' 2026-02-18T06:21:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:27.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2026-02-18T06:21:27.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING142464252mirror.0trial13020.2.0-670-gadc19233trueOK42534257mirror.1trial13020.2.0-670-gadc19233falseOK42594266mirror.2trial13020.2.0-670-gadc19233falseOK42684275mirror.3trial13020.2.0-670-gadc19233falseOKtest089f4760-8ec0-4891-8cf5-df62c4bbf216up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42464252mirror.0trial1302026-02-18 06:21:25' 2026-02-18T06:21:27.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:21:27.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:21:27.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:21:25' 2026-02-18T06:21:27.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:21:27.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:21:27.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:21:27.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:21:27.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:21:27.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:27.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror/ns1 mirror test 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:21:27.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:27.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:21:27.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror/ns1-test.export 2026-02-18T06:21:27.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror/ns1-test.export 2026-02-18T06:21:27.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:27.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:21:27.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:21:27.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:21:27.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:21:27.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/ns1/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:21:27.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:21:27.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:21:27.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:21:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test 2026-02-18T06:21:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:21:27.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:27.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test 2026-02-18T06:21:27.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:27.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:27.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:27.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2026-02-18T06:21:27.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test 2026-02-18T06:21:27.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:21:27.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror/ns1 test deleted 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns1 test 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:27.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:27.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2026-02-18T06:21:27.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:21:28.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:21:28.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:21:28.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:21:28.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:21:28.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror/ns1 test deleted '' 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns1 test 2026-02-18T06:21:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2026-02-18T06:21:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:21:28.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:21:28.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:21:28.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:28.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:21:28.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:28.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool disable mirror/ns1 2026-02-18T06:21:28.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror init-only 2026-02-18T06:21:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: mirror from non-default namespace to default namespace' 2026-02-18T06:21:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:21:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:21:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:21:28 TEST: mirror from non-default namespace to default namespace 2026-02-18T06:21:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:21:28 TEST: mirror from non-default namespace to default namespace 2026-02-18T06:21:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool --remote-namespace '' 2026-02-18T06:21:28.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror pool --remote-namespace ns1 2026-02-18T06:21:28.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns1 test journal 2026-02-18T06:21:28.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:28.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror/ns1 test 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:28.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:28.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:28.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:21:28.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:28.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns1/test 2026-02-18T06:21:28.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror/ns1 2026-02-18T06:21:28.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:28.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:28.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:29.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:21:29.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:29.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:29.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:29.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:21:29.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:21:29.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:21:29.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:29.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:29.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:29.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:29.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:29.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:29.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:29.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:29.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:29.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:21:29.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:29.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:29.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:29.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:29.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:29.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:21:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:21:29.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:29.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:29.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:29.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:29.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:21:30.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:30.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:30.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:30.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:30.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:30.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:30.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:21:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:21:30.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:30.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:30.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:30.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:21:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:21:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:30.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:30.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:21:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:32.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:32.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:32.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:32.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:32.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:32.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:32.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:32.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:32.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:32.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:32.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:32.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:32.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:21:32.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:32.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:32.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:32.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:32.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:33.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:33.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:33.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:33.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:21:33.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:21:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:33.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:21:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:33.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:33.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:33.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:33.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:33.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:21:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:21:37.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:37.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:37.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:37.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:37.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:37.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:37.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:37.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:37.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:21:37.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:21:37.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:21:37.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:21:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror/ns1 test 100 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:21:37.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:21:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:21:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:21:37.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:21:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:21:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.5 bytes/sec: 30 MiB/s 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror/ns1 test 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:37.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror/ns1 test 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:21:37.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:37.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:37.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:21:37.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:37.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2026-02-18T06:21:37.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:37.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:21:37.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:37.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:21:37.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:37.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:21:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:37.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:37.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:37.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:37.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:21:37.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2026-02-18T06:21:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:21:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2026-02-18T06:21:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:21:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:37.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:21:37.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:21:37.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:37.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:21:37.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:37.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:21:37.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:37.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:21:37.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:21:37.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:21:37.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=3 2026-02-18T06:21:37.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 1 -a 101 -le 3 2026-02-18T06:21:37.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:21:37.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:37.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:21:37.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:21:37.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:37.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:37.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:37.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:37.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:37.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:37.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:37.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:37.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:37.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:21:37.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:37.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2026-02-18T06:21:37.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2026-02-18T06:21:37.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:21:37.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:37.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:21:37.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:37.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:21:37.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:37.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:37.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:37.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:37.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:21:37.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:37.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:21:37.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:37.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:21:38.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:38.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:38.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:21:38.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:38.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:21:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:21:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:38.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:21:38.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:38.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2026-02-18T06:21:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2026-02-18T06:21:38.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:21:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:21:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:21:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:38.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:38.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:21:38.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:38.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:38.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:38.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:21:38.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:38.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:21:38.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:38.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:38.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:39.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:21:39.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:39.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:39.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:39.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:39.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:39.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:39.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:39.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:39.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:21:39.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:21:39.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b62a619a-35a7-4e59-9694-48f456ae2a11 2026-02-18T06:21:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:21:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:21:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:21:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:21:35' 2026-02-18T06:21:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:21:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:21:39.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:21:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:21:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:21:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:21:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:21:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:21:39.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:21:39.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:21:39.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:21:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:21:35' 2026-02-18T06:21:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:21:35' 2026-02-18T06:21:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:21:35' 2026-02-18T06:21:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:21:39.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING142464252mirror.0trial13020.2.0-670-gadc19233trueOK42534257mirror.1trial13020.2.0-670-gadc19233falseOK42594266mirror.2trial13020.2.0-670-gadc19233falseOK42684275mirror.3trial13020.2.0-670-gadc19233falseOKtestb62a619a-35a7-4e59-9694-48f456ae2a11up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42464252mirror.0trial1302026-02-18 06:21:35' 2026-02-18T06:21:39.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:21:39.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:21:39.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:21:35' 2026-02-18T06:21:39.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:21:39.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:21:39.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:21:39.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:21:39.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror/ns1 test 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:39.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror/ns1-test.export 2026-02-18T06:21:39.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror/ns1-test.export 2026-02-18T06:21:39.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:39.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export 2026-02-18T06:21:39.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:39.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:39.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:39.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/ns1/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export 2026-02-18T06:21:39.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:21:39.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:21:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:39.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror/ns1 test 2026-02-18T06:21:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:21:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror/ns1 test 2026-02-18T06:21:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:39.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:21:39.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:39.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test 2026-02-18T06:21:39.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test 2026-02-18T06:21:39.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:21:39.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:39.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test deleted 2026-02-18T06:21:39.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:39.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:39.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:21:39.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:21:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:21:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted '' 2026-02-18T06:21:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:21:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:21:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:21:40.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:21:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:21:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:21:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:40.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror init-only 2026-02-18T06:21:40.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool disable mirror/ns1 2026-02-18T06:21:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror pool 2026-02-18T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror pool 2026-02-18T06:21:40.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2026-02-18T06:21:40.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2026-02-18T06:21:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: add image and test replay' 2026-02-18T06:21:40.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:21:40.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:21:40.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:21:40 TEST: add image and test replay 2026-02-18T06:21:40.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:21:40 TEST: add image and test replay 2026-02-18T06:21:40.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test 2026-02-18T06:21:40.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:40.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2026-02-18T06:21:40.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:21:40.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:21:40.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:21:40.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:21:40.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_image_meta cluster2 mirror test key1 value1 2026-02-18T06:21:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=key1 2026-02-18T06:21:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=value1 2026-02-18T06:21:40.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key1 value1 2026-02-18T06:21:40.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_image_meta cluster2 mirror test key2 value2 2026-02-18T06:21:40.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:40.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:40.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=key2 2026-02-18T06:21:40.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=value2 2026-02-18T06:21:40.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key2 value2 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:40.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:21:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:21:41.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:41.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:41.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:41.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:41.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:41.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:41.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:41.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:41.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:41.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:41.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:41.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:41.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:41.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:21:41.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:41.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:41.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:21:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:41.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:41.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:41.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:41.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:41.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:41.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:41.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:21:41.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:41.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:41.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:41.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:41.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:21:41.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:41.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:41.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:41.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:41.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:21:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:41.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:41.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:21:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:42.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:42.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:42.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:42.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:42.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:42.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:42.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:42.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:42.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:42.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:42.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:42.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:42.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:42.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:42.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:42.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:42.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:21:42.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:42.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:42.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:42.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:42.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:42.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:42.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:42.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:21:42.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:42.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:42.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:42.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:42.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:42.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:42.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:21:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:21:44.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:44.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:44.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:44.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:44.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:44.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:44.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:44.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:44.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:44.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:44.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:44.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:44.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:44.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:44.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:44.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:44.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:44.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:44.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:44.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:21:44.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:44.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:44.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:21:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:21:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:21:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:44.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:44.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:44.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:44.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:44.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:44.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:21:44.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:21:44.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:44.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:44.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:44.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:21:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:49.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:49.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:49.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:49.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:21:49.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:21:49.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:21:49.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:21:49.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:21:49.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:21:49.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:21:49.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test 100 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:21:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:21:49.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:21:49.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:21:49.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 6666.83 bytes/sec: 26 MiB/s 2026-02-18T06:21:49.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:21:49.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:49.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:49.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:21:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:21:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:21:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:49.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:21:49.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:21:49.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:49.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:49.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:49.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:49.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:49.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:21:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:49.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:21:49.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:49.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:49.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:49.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:49.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:49.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2026-02-18T06:21:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2026-02-18T06:21:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:49.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:21:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:21:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:21:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:21:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:21:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:21:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:21:49.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:49.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:49.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:49.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:49.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2026-02-18T06:21:49.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2026-02-18T06:21:49.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:21:49.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:49.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:21:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:50.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:21:50.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:21:50.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:21:50.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:21:50.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2026-02-18T06:21:50.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:21:50.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:21:45' 2026-02-18T06:21:50.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:21:50.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:21:50.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:21:50.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2026-02-18T06:21:50.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:50.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:21:50.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:21:50.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:21:50.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:21:50.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:21:50.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:21:50.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:21:45' 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:21:45' 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:21:45' 2026-02-18T06:21:50.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING142464252mirror.0trial13020.2.0-670-gadc19233trueOK42534257mirror.1trial13020.2.0-670-gadc19233falseOK42594266mirror.2trial13020.2.0-670-gadc19233falseOK42684275mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}42464252mirror.0trial1302026-02-18 06:21:45' 2026-02-18T06:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:21:50.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:21:45' 2026-02-18T06:21:50.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:21:50.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:21:50.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:21:50.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:21:50.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:21:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:21:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test down+unknown 2026-02-18T06:21:50.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=down+unknown 2026-02-18T06:21:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:21:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:50.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:51.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test down+unknown '' '' 2026-02-18T06:21:51.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:51.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:51.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:51.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=down+unknown 2026-02-18T06:21:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:21:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:21:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:21:45' 2026-02-18T06:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*down+unknown' 2026-02-18T06:21:51.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: down+unknown 2026-02-18T06:21:51.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:21:51.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: status not found 2026-02-18T06:21:51.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:51.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo down+unknown 2026-02-18T06:21:51.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:21:51.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: ' 2026-02-18T06:21:51.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:21:51.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update= 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test down+unknown '' 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=down+unknown 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local prev_last_update= 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:21:51.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='UNKNOWNUNKNOWNOK1test5443312a-aa9f-4b78-8205-7ccde708800bdown+unknownstatus not foundcluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}2026-02-18 06:21:45' 2026-02-18T06:21:51.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:21:51.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:21:51.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update= 2026-02-18T06:21:51.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:21:51.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=down+unknown 2026-02-18T06:21:51.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo down+unknown 2026-02-18T06:21:51.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep down+unknown 2026-02-18T06:21:51.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:down+unknown 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:51.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:51.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:21:51.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:21:51.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:51.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:21:51.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:21:51.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:21:51.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:51.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:51.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:51.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:21:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:51.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_image_meta cluster1 mirror test key1 value1 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=key1 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local value=value1 2026-02-18T06:21:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key1 2026-02-18T06:21:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test value1 = value1 2026-02-18T06:21:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_image_meta cluster1 mirror test key2 value2 2026-02-18T06:21:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:21:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=key2 2026-02-18T06:21:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local value=value2 2026-02-18T06:21:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key2 2026-02-18T06:21:51.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test value2 = value2 2026-02-18T06:21:51.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: stop mirror, add image, start mirror and test replay' 2026-02-18T06:21:51.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:21:51.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:21:51 TEST: stop mirror, add image, start mirror and test replay 2026-02-18T06:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:21:51 TEST: stop mirror, add image, start mirror and test replay 2026-02-18T06:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster1 2026-02-18T06:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:51.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:21:51.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:51.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:51.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:0 '' 2026-02-18T06:21:51.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:51.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:21:51.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:51.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:0 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:0 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:51.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:21:51.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:21:51.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:51.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:51.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:21:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:21:51.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:21:51.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=30791 2026-02-18T06:21:51.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 30791 ']' 2026-02-18T06:21:51.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 30791 2026-02-18T06:21:51.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:21:51.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:51.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:51.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30791 '$2 == pid {print; exit 1}' 2026-02-18T06:21:51.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:21:51.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:51.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30791 '$2 == pid {print; exit 1}' 2026-02-18T06:21:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:52.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:52.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:52.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:52.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:0 2026-02-18T06:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:21:52.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:21:52.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:21:52.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:1 '' 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:1 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:1 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:52.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:52.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:52.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:21:52.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:21:52.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:21:52.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=30826 2026-02-18T06:21:52.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 30826 ']' 2026-02-18T06:21:52.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 30826 2026-02-18T06:21:52.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:21:52.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:52.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30826 '$2 == pid {print; exit 1}' 2026-02-18T06:21:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:21:52.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:52.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30826 '$2 == pid {print; exit 1}' 2026-02-18T06:21:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:52.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:52.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:52.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:52.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:52.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:52.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:52.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:52.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:1 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:52.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:52.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:52.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:52.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:52.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:21:52.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:21:52.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:21:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:2 '' 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:2 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:2 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:21:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:52.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:21:52.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:21:52.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:21:52.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=30896 2026-02-18T06:21:52.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 30896 ']' 2026-02-18T06:21:52.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 30896 2026-02-18T06:21:52.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:21:52.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:52.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:52.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30896 '$2 == pid {print; exit 1}' 2026-02-18T06:21:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:21:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:52.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30896 '$2 == pid {print; exit 1}' 2026-02-18T06:21:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:21:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:52.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:52.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:52.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:52.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:2 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:52.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:21:52.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:21:52.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:21:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:3 '' 2026-02-18T06:21:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:21:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:21:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:21:52.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:3 2026-02-18T06:21:52.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:3 2026-02-18T06:21:52.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:52.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:21:52.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:52.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:52.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:21:52.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:21:52.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:21:52.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=30972 2026-02-18T06:21:52.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 30972 ']' 2026-02-18T06:21:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 30972 2026-02-18T06:21:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:21:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30972 '$2 == pid {print; exit 1}' 2026-02-18T06:21:52.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:21:52.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:21:52.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=30972 '$2 == pid {print; exit 1}' 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:52.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:52.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:52.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:52.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:3 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:52.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:52.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:52.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:52.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:52.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:52.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:21:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:21:52.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:21:52.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image1=test1 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test1 journal 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:21:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test1 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:21:52.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test1 2026-02-18T06:21:52.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:21:52.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:52.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test1 100 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:21:52.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test1 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:21:52.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:21:52.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:21:52.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7143.03 bytes/sec: 28 MiB/s 2026-02-18T06:21:52.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster1 2026-02-18T06:21:52.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:52.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:0 2026-02-18T06:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:52.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:52.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:1 2026-02-18T06:21:52.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:21:52.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:52.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:52.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:2 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:52.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:52.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:3 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:21:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:21:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:21:52.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-18T06:21:52.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:52.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:52.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:52.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-18T06:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:52.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-18T06:21:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:21:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:21:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:21:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:21:53.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:53.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:53.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:21:53.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:53.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:53.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:53.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:53.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2026-02-18T06:21:53.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:53.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:53.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test1 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:53.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:53.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:53.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:53.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test1 2026-02-18T06:21:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:53.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test1 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:53.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:53.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test1 2026-02-18T06:21:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:21:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:21:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:21:54.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:21:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:21:54.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2026-02-18T06:21:54.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:54.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:54.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test1 2026-02-18T06:21:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:21:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:21:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:21:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:21:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test1 2026-02-18T06:21:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:54.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test1 2026-02-18T06:21:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:21:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:21:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:54.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:21:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:21:54.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:21:54.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:54.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:54.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:21:54.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test1 2026-02-18T06:21:54.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-18T06:21:54.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:21:54.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:21:54.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:54.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:21:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:56.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:21:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:21:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:56.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:56.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:56.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:56.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:21:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:21:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test1 2026-02-18T06:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test1 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:21:57.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:21:57.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:21:57.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test1 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test1 ']' 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test1' 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test1 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:21:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:21:57.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test1 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:21:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:21:57.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:21:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:21:57.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:57.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:57.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:21:57.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test1 2026-02-18T06:21:57.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:57.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test1 2026-02-18T06:21:57.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:57.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test1 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test1 '' 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test1 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test1.status 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test1.status 2026-02-18T06:21:57.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:57.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.status 2026-02-18T06:21:57.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1 2026-02-18T06:21:57.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.status 2026-02-18T06:21:57.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:57.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:57.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:57.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:21:57.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:21:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.status 2026-02-18T06:21:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:21:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test1 2026-02-18T06:21:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test1 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test1 '..*' 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test1 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test1.status 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test1.status 2026-02-18T06:21:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:21:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.status 2026-02-18T06:21:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1 2026-02-18T06:21:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.status 2026-02-18T06:21:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:21:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:21:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:21:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:21:57.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:21:57.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.status 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test1 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:57.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:57.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:57.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:57.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:57.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:57.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:57.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:21:58.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:21:58.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:21:58.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:21:58.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:21:58.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2026-02-18T06:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:00.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:00.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:00.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:00.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:00.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:00.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:22:00.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:22:00.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:22:00.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:00.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:22:00.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:00.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:22:04.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:04.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:22:04.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:22:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:04.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:22:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2026-02-18T06:22:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:22:12.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:12.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:12.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:12.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:12.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:12.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:12.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:22:12.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:22:12.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:22:12.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:12.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:22:12.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:12.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:22:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2026-02-18T06:22:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:22:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:22:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:22:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:22:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:22:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:22:28.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2026-02-18T06:22:28.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:28.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:28.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:28.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:28.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:22:28.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:28.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:28.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:28.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:28.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:28.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:28.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:28.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":17.5,"entries_behind_primary":0,"entries_per_second":0.2,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:28.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:28.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:22' 2026-02-18T06:22:28.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:28.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:28.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:22:28.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":17.5,"entries_behind_primary":0,"entries_per_second":0.2,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:28.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:28.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:28.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:28.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:28.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:28.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:28.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:28.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:28.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:22' 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-18 06:22:22' 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:22' 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:28.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}43444348mirror.0trial1302026-02-18 06:22:22test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":17.5,"entries_behind_primary":0,"entries_per_second":0.2,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:22' 2026-02-18T06:22:28.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:28.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:22:28.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:22' 2026-02-18T06:22:28.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:22:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:22:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test1 down+unknown 2026-02-18T06:22:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:28.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=down+unknown 2026-02-18T06:22:28.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:28.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:28.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:28.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test1 down+unknown '' '' 2026-02-18T06:22:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=down+unknown 2026-02-18T06:22:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test1 2026-02-18T06:22:29.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":17.5,"entries_behind_primary":0,"entries_per_second":0.2,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:22' 2026-02-18T06:22:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*down+unknown' 2026-02-18T06:22:29.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: down+unknown 2026-02-18T06:22:29.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:29.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: status not found 2026-02-18T06:22:29.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:29.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo down+unknown 2026-02-18T06:22:29.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:29.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: ' 2026-02-18T06:22:29.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:29.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update= 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test1 down+unknown '' 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=down+unknown 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local prev_last_update= 2026-02-18T06:22:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='UNKNOWNUNKNOWNOK2test5443312a-aa9f-4b78-8205-7ccde708800bdown+unknownstatus not foundcluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}2026-02-18 06:22:22test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86edown+unknownstatus not foundcluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":17.5,"entries_behind_primary":0,"entries_per_second":0.2,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2026-02-18 06:22:22' 2026-02-18T06:22:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:22:29.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update= 2026-02-18T06:22:29.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:22:29.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=down+unknown 2026-02-18T06:22:29.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo down+unknown 2026-02-18T06:22:29.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep down+unknown 2026-02-18T06:22:29.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:down+unknown 2026-02-18T06:22:29.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:29.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:29.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test1 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test1.export 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test1.export 2026-02-18T06:22:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:29.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.export 2026-02-18T06:22:29.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test1.export 2026-02-18T06:22:29.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test1.export 2026-02-18T06:22:29.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:29.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test1.export 2026-02-18T06:22:29.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test1.export 2026-02-18T06:22:29.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test1 /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.export 2026-02-18T06:22:29.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:22:29.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test1 /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test1.export 2026-02-18T06:22:29.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:22:29.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test1.export 2026-02-18T06:22:29.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test1.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test1.export 2026-02-18T06:22:29.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:29.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: test the first image is replaying after restart' 2026-02-18T06:22:29.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:22:29.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:22:29.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:22:29 TEST: test the first image is replaying after restart 2026-02-18T06:22:29.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:22:29 TEST: test the first image is replaying after restart 2026-02-18T06:22:29.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test 100 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:22:29.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:22:29.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:22:29.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.44 bytes/sec: 30 MiB/s 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:22:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:30.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:30.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:22:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:22:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:22:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:31.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:31.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:31.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:31.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:31.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:31.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:22:31.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:31.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:22:31.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:22:31.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:31.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:22:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:22:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:22:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:22:31.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:22:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:22:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:22:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:22:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:22:31.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:22:31.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:22:31.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:31.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:22:31.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:31.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:22:31.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:31.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:22:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:31.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:22:31.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:22:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:22:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:22:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:22:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:22:31.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:31.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:22:31.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:22:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:22:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:22:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:22:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:22:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:22:31.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:22:31.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:22:31.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:22:31.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:22:31.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:22:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:22:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=101' '!=' '' 2026-02-18T06:22:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:22:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=101' = '' 2026-02-18T06:22:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:22:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:22:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:22:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:22:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=5 2026-02-18T06:22:31.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:22:31.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:22:31.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:22:31.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=4 2026-02-18T06:22:31.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:22:31.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:22:31.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:22:31.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:22:31.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2026-02-18T06:22:31.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:22:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:22:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=101 2026-02-18T06:22:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 5 = 4 -a 101 -le 101 2026-02-18T06:22:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:22:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:22:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:22:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:22:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:22:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:22:31.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:31.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:31.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:31.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:22:31.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:22:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:22:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:22:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:22:31.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:22:31.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:22:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:31.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:22:31.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:22:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:22:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:22:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:22:31.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:32.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:22:32.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:32.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:32.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:22:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:32.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:22:32.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:22:32.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:22:32.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:32.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2026-02-18T06:22:32.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:32.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:22' 2026-02-18T06:22:32.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:32.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:32.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:22:32.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2026-02-18T06:22:32.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:32.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:32.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:32.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:32.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:32.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:32.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:32.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:22' 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:22:22' 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:22' 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:32.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:32.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}43444348mirror.0trial1302026-02-18 06:22:22test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":17.5,"entries_behind_primary":0,"entries_per_second":0.2,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:22' 2026-02-18T06:22:32.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:32.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:22:32.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:22' 2026-02-18T06:22:32.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:22:32.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:32.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:32.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:32.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:32.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:22:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:22:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:22:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:22:32.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:22:32.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:22:32.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:22:32.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:22:32.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:22:32.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:22:32.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:22:32.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:22:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:22:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:22:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: stop/start/restart mirror via admin socket' 2026-02-18T06:22:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:22:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:22:33.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:22:33 TEST: stop/start/restart mirror via admin socket 2026-02-18T06:22:33.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:22:33 TEST: stop/start/restart mirror via admin socket 2026-02-18T06:22:33.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2026-02-18T06:22:33.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:33.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:33.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2026-02-18T06:22:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2026-02-18T06:22:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:22:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:22:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:22:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:22:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:22:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:22:33.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:33.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:33.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:33.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2026-02-18T06:22:33.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror stop 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:33.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:22:33.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:22:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:22:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop 2026-02-18T06:22:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror stop 2026-02-18T06:22:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:22:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:22:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:22:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:22:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:22:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:22:33.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:33.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:33.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:33.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop 2026-02-18T06:22:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:22:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:34.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:34.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:34.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:34.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:34.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:34.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:22:34.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:34.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:34.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:34.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:34.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:34.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:34.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:34.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:34.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:34.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:22:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:22:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:22:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:34.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:34.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:22:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:35.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:35.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:35.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:35.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:22:35.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:35.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:22:35.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:22:35.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:35.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:22:35.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:35.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:22:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:22:36.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:22:36.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:22:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:22:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:22:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:33' 2026-02-18T06:22:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:22:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:22:36.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:36.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:22:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:36.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:36.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:33' 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:22:33' 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:33' 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:36.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:33test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:33' 2026-02-18T06:22:36.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:36.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:22:36.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:33' 2026-02-18T06:22:36.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:22:36.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:22:36.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:36.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:22:36.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:36.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:36.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:36.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:37.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-18T06:22:37.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:37.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:37.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:37.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:37.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:37.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:37.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:37.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:37.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:22:37.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:22:37.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:37.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:33' 2026-02-18T06:22:37.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:22:37.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:22:37.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:37.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:22:37.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:37.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:33' 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-18 06:22:33' 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:33' 2026-02-18T06:22:37.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:37.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:37.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:37.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:33test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:33' 2026-02-18T06:22:37.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:37.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:22:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:33' 2026-02-18T06:22:37.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:22:37.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:22:37.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:37.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:22:37.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror start 2026-02-18T06:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror start 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:37.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:37.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:37.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start 2026-02-18T06:22:37.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror start 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:22:37.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:37.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:22:37.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:22:37.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:37.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:37.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:37.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start 2026-02-18T06:22:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror start 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:37.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:22:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:22:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:22:37.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:37.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:37.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:37.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror start 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:37.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:22:37.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:22:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:22:37.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:37.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:37.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:37.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start 2026-02-18T06:22:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:22:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:37.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:38.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:22:38.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:38.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:22:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:38.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:22:38.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:38.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:38.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:38.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:38.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:38.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:38.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:38.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:38.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:22:38.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:38.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:22:38.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:22:38.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:38.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:22:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:22:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:22:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-18T06:22:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:39.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-18T06:22:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:22:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:39.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:22:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:39.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:39.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:39.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:39.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:22:39.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:39.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:22:39.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:22:39.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:39.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:22:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:22:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:22:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:22:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:22:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:37' 2026-02-18T06:22:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:22:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:41.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:41.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:41.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:41.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:41.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:41.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:41.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:41.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:37' 2026-02-18T06:22:41.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:22:37' 2026-02-18T06:22:41.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:41.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:41.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:41.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:41.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:37' 2026-02-18T06:22:41.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:41.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:41.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:41.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:22:37test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:37' 2026-02-18T06:22:41.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:41.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:22:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:37' 2026-02-18T06:22:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:22:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:41.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:41.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:41.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:41.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:37' 2026-02-18T06:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:37' 2026-02-18T06:22:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-18 06:22:37' 2026-02-18T06:22:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:42.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:37' 2026-02-18T06:22:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:42.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:42.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:22:37test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:37' 2026-02-18T06:22:42.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:42.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:22:42.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:37' 2026-02-18T06:22:42.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:22:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:42.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:42.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:42.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2026-02-18T06:22:42.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2026-02-18T06:22:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:42.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:42.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:42.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:42.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:42.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:42.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2026-02-18T06:22:42.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:42.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2026-02-18T06:22:42.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:22:42.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:22:42.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:42.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:42.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:22:42.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:22:42.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:22:42.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:42.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:42.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:42.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2026-02-18T06:22:42.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:42.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror restart 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:42.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:22:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:22:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:22:42.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:42.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:42.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:42.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart 2026-02-18T06:22:42.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror restart 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:22:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:22:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:22:42.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart 2026-02-18T06:22:42.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:42.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:43.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:22:43.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:43.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:43.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:43.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:43.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:43.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:43.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:43.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:43.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:22:43.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:43.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:22:43.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:22:43.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:43.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-18T06:22:43.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:43.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:43.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:43.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:22:43.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:43.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:43.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:44.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-18T06:22:44.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:44.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:44.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:44.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:22:44.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:44.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:44.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:44.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:44.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:44.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:44.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:44.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:22:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:22:44.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:22:44.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:44.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:44.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:44.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:45.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:22:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:22:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:22:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:22:45.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:45.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:42' 2026-02-18T06:22:45.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:45.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:45.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:45.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:22:45.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:45.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:45.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:45.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:45.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:45.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:45.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:45.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:42' 2026-02-18T06:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:22:42' 2026-02-18T06:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:42' 2026-02-18T06:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:45.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:22:42test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:42' 2026-02-18T06:22:45.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:45.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:22:45.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:42' 2026-02-18T06:22:45.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:22:45.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:45.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:45.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:45.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:45.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:45.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:46.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-18T06:22:46.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:42' 2026-02-18T06:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:46.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:46.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:46.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:46.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:46.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:46.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:42' 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-18 06:22:42' 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:42' 2026-02-18T06:22:46.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:46.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:46.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:22:42test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:42' 2026-02-18T06:22:46.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:46.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:22:46.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:42' 2026-02-18T06:22:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:22:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2026-02-18T06:22:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2026-02-18T06:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2026-02-18T06:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:46.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:22:46.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:22:46.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:22:46.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:46.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror stop 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:46.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:22:46.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:22:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop 2026-02-18T06:22:46.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:46.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror stop 2026-02-18T06:22:46.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:22:46.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:46.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:46.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:46.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop 2026-02-18T06:22:46.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:22:46.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:46.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:46.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:46.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:22:46.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:46.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:46.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:46.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:22:46.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:46.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:46.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:22:47.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:22:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:47.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:47.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:22:47.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:47.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:47.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:47.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:47.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:22:47.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:47.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:22:47.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:22:47.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:47.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:22:47.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:47.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:47.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:22:48.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:22:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:48.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:22:48.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:48.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:48.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:48.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:48.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:48.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:48.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:48.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:22:49.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:49.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:22:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:22:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:49.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:50.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:50.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:22:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:22:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:22:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:22:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:22:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:46' 2026-02-18T06:22:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:22:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:22:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:22:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:50.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:50.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:46' 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:22:46' 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:46' 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:46test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:46' 2026-02-18T06:22:50.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:50.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:22:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:46' 2026-02-18T06:22:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:22:50.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:22:50.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:50.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:22:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-18T06:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-18T06:22:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:51.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:51.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:51.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:51.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:51.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:51.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:51.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:22:51.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:22:51.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:51.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:46' 2026-02-18T06:22:51.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:22:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:22:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:22:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:51.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:51.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:46' 2026-02-18T06:22:51.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-18 06:22:46' 2026-02-18T06:22:51.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:51.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:46' 2026-02-18T06:22:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:46test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:46' 2026-02-18T06:22:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:22:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:46' 2026-02-18T06:22:51.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:22:51.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:22:51.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:51.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:22:51.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:51.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:51.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:51.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2026-02-18T06:22:51.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:51.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:51.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:51.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:51.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2026-02-18T06:22:51.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:51.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:51.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:22:51.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:22:51.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:22:51.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:22:51.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:22:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2026-02-18T06:22:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror restart 2026-02-18T06:22:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:22:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:22:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:22:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:22:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart 2026-02-18T06:22:51.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:51.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror restart 2026-02-18T06:22:51.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:51.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:22:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:22:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:22:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart 2026-02-18T06:22:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:22:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:22:51.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:22:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:51.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:22:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:52.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:22:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:22:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:52.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:52.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:52.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:52.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:52.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:52.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:52.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:22:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:52.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:52.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:52.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:52.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:52.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:22:52.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:22:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:22:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:22:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:22:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:22:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:52.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:53.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:53.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:53.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:22:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:53.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:53.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:53.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:53.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:53.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:53.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:53.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:53.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:53.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:22:53.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:53.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:22:53.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:22:53.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:53.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:22:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:22:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:22:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:22:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:53.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:53.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:53.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:53.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:53.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:53.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:54.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:52' 2026-02-18T06:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:54.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:54.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:54.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:22:54.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:54.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:54.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:54.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:54.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:54.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:54.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:54.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:54.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:52' 2026-02-18T06:22:54.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:22:52' 2026-02-18T06:22:54.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:54.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:54.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:54.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:54.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:52' 2026-02-18T06:22:54.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:54.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:54.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:54.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:22:52test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:52' 2026-02-18T06:22:54.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:54.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:22:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:52' 2026-02-18T06:22:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:22:54.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:54.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:52' 2026-02-18T06:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:22:55.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:22:55.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:55.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:22:55.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:55.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:55.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:55.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:55.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:55.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:55.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:55.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:52' 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-18 06:22:52' 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:52' 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:55.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:22:52test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:22:52' 2026-02-18T06:22:55.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:55.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:22:55.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:52' 2026-02-18T06:22:55.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:22:55.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:22:55.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:22:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:22:55.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:22:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:22:55.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:55.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:55.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:55.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:55.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:55.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:22:55.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2026-02-18T06:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2 2026-02-18T06:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:22:55.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:22:55.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:55.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:22:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2 2026-02-18T06:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2 2026-02-18T06:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:22:55.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:55.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:55.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:22:55.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:55.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:22:55.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:55.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:55.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:56.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:22:56.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:56.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:56.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:56.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:56.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:22:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:56.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:56.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:56.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:56.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:56.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:22:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:22:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:22:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:22:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:22:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:58.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:22:58.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:22:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:22:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:22:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:22:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:22:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:22:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:22:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:22:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:22:58.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:58.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:58.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:22:58.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:22:58.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:22:58.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:22:58.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:22:58.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:22:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:22:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:58.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:22:59.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:22:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:22:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:22:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:22:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:22:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:22:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:55' 2026-02-18T06:22:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:22:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:22:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:22:59.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:22:59.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:22:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:22:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:22:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:22:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:22:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:22:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:55' 2026-02-18T06:22:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:22:55' 2026-02-18T06:22:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:59.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:22:59.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:59.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:55' 2026-02-18T06:22:59.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:22:59.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:22:59.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:22:59.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:55test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:55' 2026-02-18T06:22:59.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:22:59.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:22:59.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:55' 2026-02-18T06:22:59.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:22:59.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:22:59.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:22:59.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:22:59.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:22:59.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:22:59.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:22:59.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:00.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-18T06:23:00.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:00.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:00.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:23:00.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:22:55' 2026-02-18T06:23:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:00.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:23:00.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:00.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:00.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:00.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:00.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:00.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:00.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:00.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:55' 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-18 06:22:55' 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:22:55' 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:00.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:00.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:55test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:55' 2026-02-18T06:23:00.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:00.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:23:00.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:22:55' 2026-02-18T06:23:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:23:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:23:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:23:00.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:00.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:00.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:00.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:00.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:00.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:00.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:00.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:00.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:00.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:00.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:00.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:00.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:00.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:00.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2026-02-18T06:23:00.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:00.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:00.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:00.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:01.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:01.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:01.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:01.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:01.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:01.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:01.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:01.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:01.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:01.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:01.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:01.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:01.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:01.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:01.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:01.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:01.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:01.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:01.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:01.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:02.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:23:02.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:02.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:02.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:02.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:02.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:02.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:02.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:02.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:02.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:02.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:02.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:23:02.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:02.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:02.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:02.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:00' 2026-02-18T06:23:02.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:02.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:02.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:02.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:02.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:02.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:02.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:02.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:02.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:02.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:02.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:02.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:02.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:00' 2026-02-18T06:23:02.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:23:00' 2026-02-18T06:23:02.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:02.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:02.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:02.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:02.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:00' 2026-02-18T06:23:02.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:02.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:02.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:02.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:23:00test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:22:55' 2026-02-18T06:23:02.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:02.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:02.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:00' 2026-02-18T06:23:02.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:02.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:02.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:02.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:02.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:02.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:02.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror start mirror cluster2 2026-02-18T06:23:02.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:02.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:02.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:02.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:02.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror cluster2 2026-02-18T06:23:02.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:02.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:02.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:02.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:02.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:02.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:02.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:02.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:02.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:02.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:02.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror cluster2 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror cluster2 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:02.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:02.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:02.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:02.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:02.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror cluster2 2026-02-18T06:23:02.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:02.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror start mirror cluster2 2026-02-18T06:23:02.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:23:02.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:02.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:02.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:23:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:23:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:02.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:02.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:02.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:02.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:02.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:02.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:02.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start mirror cluster2 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror start mirror cluster2 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:23:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:02.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:02.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:02.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start mirror cluster2 2026-02-18T06:23:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-18T06:23:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:03.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:03.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:03.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:03.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:03.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:03.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:03.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:23:03.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:03.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:03.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:03.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:03.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-18T06:23:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:04.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:04.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:23:04.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:23:04.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:23:04.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:04.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:23:04.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:04.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:02' 2026-02-18T06:23:04.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:04.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:04.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:04.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:23:04.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:04.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:04.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:04.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:04.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:04.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:04.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:04.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:04.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:02' 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-18 06:23:02' 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:02' 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:04.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:23:00test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:23:02' 2026-02-18T06:23:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:04.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:23:04.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:02' 2026-02-18T06:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:04.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:04.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:04.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:04.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:04.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test 2026-02-18T06:23:04.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:04.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:04.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:04.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:04.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:04.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test 2026-02-18T06:23:04.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:04.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:05.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:05.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:05.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:05.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:05.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:05.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:05.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:05.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:05.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:05.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:05.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:05.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:23:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:04' 2026-02-18T06:23:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:06.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:07.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:07.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:07.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:07.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:07.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:07.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:04' 2026-02-18T06:23:07.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:23:04' 2026-02-18T06:23:07.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:07.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:07.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:07.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:07.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:04' 2026-02-18T06:23:07.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:07.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:07.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:23:04test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:23:02' 2026-02-18T06:23:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:07.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:04' 2026-02-18T06:23:07.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:07.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:07.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:07.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:07.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:07.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:07.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2026-02-18T06:23:07.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:07.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:07.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2026-02-18T06:23:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:07.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:07.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:07.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:07.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:07.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:07.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:07.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:07.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:07.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:07.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:07.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:07.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:07.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:07.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:07.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:07.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:07.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:07.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2 2026-02-18T06:23:07.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:23:07.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:07.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:07.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:07.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:07.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:07.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:07.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:07.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:07.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:07.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:07.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:23:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:07.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:07.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:07.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:07.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:07.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:07.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:07.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:08.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:08.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:08.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:08.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:08.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:08.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:08.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:08.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:08.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:08.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:08.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:08.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-18T06:23:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:09.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:09.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:23:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:09.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:09.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:09.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:09.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:09.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:09.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:09.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:09.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:09.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:09.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:23:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:09.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:09.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:09.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:09.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:09.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:09.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:09.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2 2026-02-18T06:23:09.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:23:09.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:09.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:09.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:10.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:23:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:10.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:10.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:10.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:10.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:10.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:10.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:10.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:10.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:10.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:10.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:10.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:10.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:10.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:23:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:23:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:10.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:23:10.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:10.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-18T06:23:10.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:10.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:10.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:10.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-18T06:23:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:23:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:11.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:11.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:11.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:11.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:11.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:11.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:11.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:11.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:11.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:23:11.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:11.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:23:11.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:23:11.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:11.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:23:11.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:11.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:12.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:23:12.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:12.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:12.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:12.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:12.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:12.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:23:12.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:23:12.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:23:12.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:12.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:09' 2026-02-18T06:23:12.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:23:12.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:23:12.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:12.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:23:12.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:12.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:12.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:12.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:12.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:12.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:12.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:12.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:12.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:09' 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:23:09' 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:09' 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:12.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:23:09test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:23:09' 2026-02-18T06:23:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:12.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:09' 2026-02-18T06:23:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:23:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:23:12.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:12.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-18T06:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:13.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:13.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:13.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:09' 2026-02-18T06:23:13.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:23:13.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:23:13.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:13.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:23:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:09' 2026-02-18T06:23:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-18 06:23:09' 2026-02-18T06:23:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:09' 2026-02-18T06:23:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+stoppedstopped43444348mirror.0trial1302026-02-18 06:23:09test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:23:09' 2026-02-18T06:23:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:23:13.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:09' 2026-02-18T06:23:13.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:23:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:23:13.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:13.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:23:13.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2026-02-18T06:23:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:23:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:14.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:14.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2 2026-02-18T06:23:14.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:23:14.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:14.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2 2026-02-18T06:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:14.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:14.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:14.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:14.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:14.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:14.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:15.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:15.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:15.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:15.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:15.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:15.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:15.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:15.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:15.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:15.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:15.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:15.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:15.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:15.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:15.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:15.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:15.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:15.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-18T06:23:15.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:15.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:15.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:15.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:15.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:15.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:15.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:16.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:16.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:16.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:16.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:16.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-18T06:23:16.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:16.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:16.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:16.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:16.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:16.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:16.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:16.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:16.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:23:16.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:16.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:16.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:16.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:16.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:16.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:16.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:16.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:17.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:17.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:17.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:17.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:23:17.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:17.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:17.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:17.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:14' 2026-02-18T06:23:17.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:17.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:17.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:17.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:17.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:17.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:17.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:17.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:17.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:17.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:17.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:17.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:14' 2026-02-18T06:23:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:23:14' 2026-02-18T06:23:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:14' 2026-02-18T06:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:17.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:23:14test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:23:14' 2026-02-18T06:23:17.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:17.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:14' 2026-02-18T06:23:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:17.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:17.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:17.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:17.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:17.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:17.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:17.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-18T06:23:17.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:17.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:17.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:17.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:17.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:17.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:17.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:17.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:18.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-18T06:23:18.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-18T06:23:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test1: 2026-02-18T06:23:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 6b35e0c1-7b1f-4a1c-b955-fdf3f833c86e 2026-02-18T06:23:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:23:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:14' 2026-02-18T06:23:18.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:18.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:18.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:18.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:23:18.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:18.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:18.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:18.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:18.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:18.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:18.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:18.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:18.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:14' 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-18 06:23:14' 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:14' 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:18.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:18.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:23:14test16b35e0c1-7b1f-4a1c-b955-fdf3f833c86eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:23:14' 2026-02-18T06:23:18.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:18.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-18T06:23:18.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:14' 2026-02-18T06:23:18.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-18T06:23:18.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:18.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:18.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:18.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool= 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image= 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:18.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:18.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush 2026-02-18T06:23:18.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:18.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:18.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:18.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:18.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:18.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:18.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:18.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:18.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:18.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush 2026-02-18T06:23:18.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:18.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ all_admin_daemons cluster1 rbd mirror status 2026-02-18T06:23:18.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:18.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:18.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:18.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror status 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:18.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:18.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:18.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:18.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status 2026-02-18T06:23:18.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:23:18.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool_replayers": [ 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: d037f6ff-011b-4052-abe8-144fb9e70771 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror", 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4348", 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4348", 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": true, 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instances": [ 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4348", 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4355", 2026-02-18T06:23:18.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4365", 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4372" 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ], 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.33178.cluster1.94450691103104.asok", 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33178.cluster2.94450703210880.asok", 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [ 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "name": "mirror/test", 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "name": "mirror/test1", 2026-02-18T06:23:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ], 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter": { 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter_status": { 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "delete_images_queue": [], 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "failed_deletes_queue": [] 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [], 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter": { 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter_status": { 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "delete_images_queue": [], 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "failed_deletes_queue": [] 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns2", 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns2", 2026-02-18T06:23:18.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [], 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter": { 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter_status": { 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "delete_images_queue": [], 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "failed_deletes_queue": [] 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: 16035ff9-d0e7-45fb-8544-bc6c842315e4 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror_parent", 2026-02-18T06:23:18.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4360", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4360", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": true, 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instances": [ 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4360", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4374", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4377", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "4378" 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ], 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.33178.cluster1.94450694855040.asok", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33178.cluster2.94450697918848.asok", 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [], 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter": { 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter_status": { 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "delete_images_queue": [], 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "failed_deletes_queue": [] 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [], 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter": { 2026-02-18T06:23:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_deleter_status": { 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "delete_images_queue": [], 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "failed_deletes_queue": [] 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:23:18.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:18.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror status 2026-02-18T06:23:18.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:23:18.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:18.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:18.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:18.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:18.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status 2026-02-18T06:23:18.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:23:18.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool_replayers": [ 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: d037f6ff-011b-4052-abe8-144fb9e70771 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror", 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4355", 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4348", 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": false, 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.33221.cluster1.93976477825408.asok", 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33221.cluster2.93976490154368.asok", 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns2", 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns2", 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: 16035ff9-d0e7-45fb-8544-bc6c842315e4 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror_parent", 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4377", 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4360", 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": false, 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.33221.cluster1.93976481847680.asok", 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33221.cluster2.93976484018560.asok", 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:23:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror status 2026-02-18T06:23:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:23:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:18.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:18.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:18.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:18.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status 2026-02-18T06:23:18.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:23:18.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool_replayers": [ 2026-02-18T06:23:18.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: d037f6ff-011b-4052-abe8-144fb9e70771 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror", 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4365", 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4348", 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": false, 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.33283.cluster1.94243736435072.asok", 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33283.cluster2.94243748755840.asok", 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns2", 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns2", 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: 16035ff9-d0e7-45fb-8544-bc6c842315e4 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror_parent", 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4374", 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4360", 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": false, 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.33283.cluster1.94243740383616.asok", 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33283.cluster2.94243743218048.asok", 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:23:18.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror status 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:23:18.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:18.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:18.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:18.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:18.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:18.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:18.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:18.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:18.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:18.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:18.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:18.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status 2026-02-18T06:23:18.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:23:18.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool_replayers": [ 2026-02-18T06:23:18.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: d037f6ff-011b-4052-abe8-144fb9e70771 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror", 2026-02-18T06:23:18.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4372", 2026-02-18T06:23:18.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4348", 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": false, 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.33360.cluster1.94150559504768.asok", 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33360.cluster2.94150572628352.asok", 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns2", 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns2", 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "peer": "uuid: 16035ff9-d0e7-45fb-8544-bc6c842315e4 cluster: cluster2 client: client.mirror", 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "pool": "mirror_parent", 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "instance_id": "4378", 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "stopped (manual)", 2026-02-18T06:23:18.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader_instance_id": "4360", 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "leader": false, 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.33360.cluster1.94150563240320.asok", 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.33360.cluster2.94150565886336.asok", 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "sync_throttler": { 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 5, 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "deletion_throttler": { 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "max_parallel_requests": 1, 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "running_requests": 0, 2026-02-18T06:23:18.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "waiting_requests": 0 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "namespace_replayers": [ 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "", 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "", 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: }, 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: { 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "local_namespace": "ns1", 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "remote_namespace": "ns1", 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "image_replayers": [] 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: } 2026-02-18T06:23:18.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: ] 2026-02-18T06:23:18.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:23:18.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test1 2026-02-18T06:23:18.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:18.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:18.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:18.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:23:18.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:18.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test1 2026-02-18T06:23:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test1 2026-02-18T06:23:18.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test1 2026-02-18T06:23:18.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test1 2026-02-18T06:23:18.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:23:18.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:18.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: test image rename' 2026-02-18T06:23:18.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:23:18.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:23:18.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:23:18 TEST: test image rename 2026-02-18T06:23:18.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:23:18 TEST: test image rename 2026-02-18T06:23:18.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ new_name=test_RENAMED 2026-02-18T06:23:18.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_image cluster2 mirror test test_RENAMED 2026-02-18T06:23:18.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:18.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:18.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:18.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_name=test_RENAMED 2026-02-18T06:23:18.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rename mirror/test mirror/test_RENAMED 2026-02-18T06:23:18.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:18.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:19.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2026-02-18T06:23:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:19.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:19.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:19.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:19.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:19.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:19.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:19.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:19.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:19.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:19.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-18T06:23:19.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:19.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:23:19.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:23:19.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:19.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:19.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:19.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:19.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:20.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-18T06:23:20.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:20.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:20.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:20.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:20.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:23:20.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:20.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:20.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:20.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:23:20.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:20.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:20.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:20.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:20.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-18T06:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:23:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:20.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:20.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:20.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:20.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:20.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-18T06:23:20.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:20.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:20.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:20.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:23:21.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:21.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:21.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:21.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:21.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:21.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:21.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:21.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:21.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:21.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:21.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:21.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:21.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:21.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:22.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2026-02-18T06:23:22.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:22.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:22.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:22.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:22.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:22.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:22.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:22.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:22.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2026-02-18T06:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_RENAMED: 2026-02-18T06:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:14' 2026-02-18T06:23:22.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:22.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:22.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:22.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:23:22.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:22.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:22.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:22.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:22.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:22.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:22.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:22.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:14' 2026-02-18T06:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2026-02-18 06:23:14' 2026-02-18T06:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:22.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:14' 2026-02-18T06:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest_RENAMED5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:23:14' 2026-02-18T06:23:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:22.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2026-02-18T06:23:22.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:14' 2026-02-18T06:23:22.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2026-02-18T06:23:22.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:22.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:22.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:22.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:22.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:22.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:22.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:22.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:22.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:22.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:22.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:22.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:22.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:22.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:22.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:22.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:23:22.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "name": "mirror/test_RENAMED", 2026-02-18T06:23:22.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:22.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:23:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test_RENAMED 2026-02-18T06:23:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:22.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:22.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:22.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:22.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test_RENAMED 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:22.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:22.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test_RENAMED 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:22.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:22.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:22.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:22.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:22.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:23.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:23.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:23.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:23.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:23.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:23.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:23.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:23.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:23.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:23.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:23.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:23.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2026-02-18T06:23:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2026-02-18T06:23:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_RENAMED: 2026-02-18T06:23:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 5443312a-aa9f-4b78-8205-7ccde708800b 2026-02-18T06:23:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2026-02-18T06:23:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:22' 2026-02-18T06:23:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:24.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:24.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:24.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2026-02-18T06:23:24.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:24.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:24.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:24.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:24.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:24.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:24.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:24.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:22' 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2026-02-18 06:23:22' 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:22' 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:24.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:24.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtest_RENAMED5443312a-aa9f-4b78-8205-7ccde708800bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}}43444348mirror.0trial1302026-02-18 06:23:22' 2026-02-18T06:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2026-02-18T06:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:22' 2026-02-18T06:23:24.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2026-02-18T06:23:24.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:24.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:24.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_image cluster2 mirror test_RENAMED test 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_RENAMED 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_name=test 2026-02-18T06:23:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rename mirror/test_RENAMED mirror/test 2026-02-18T06:23:24.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:23:24.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:24.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:24.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:24.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:25.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:25.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:25.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:25.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:25.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:25.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:25.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:25.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:25.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:25.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:25.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:25.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:25.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:25.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:25.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:25.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:no valid command found; 1 closest matches: 2026-02-18T06:23:25.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status mirror/test_RENAMED 2026-02-18T06:23:25.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:25.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:23:25.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:23:25.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:25.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:25.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:25.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:25.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:23:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:25.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:25.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:25.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:23:25.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:25.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:25.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:25.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:25.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:23:25.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:25.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:25.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:25.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:25.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:23:25.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:23:25.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:25.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:25.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:25.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:25.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:23:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:26.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:26.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:26.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:26.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:26.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:26.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:26.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:26.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:26.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:26.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:26.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:26.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:26.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:26.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: test trash move restore' 2026-02-18T06:23:26.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:23:26.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:23:26.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:23:26 TEST: test trash move restore 2026-02-18T06:23:26.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:23:26 TEST: test trash move restore 2026-02-18T06:23:26.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster2 mirror test 2026-02-18T06:23:26.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:26.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:26.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:26.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster2 info mirror/test 2026-02-18T06:23:26.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:23:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id=112f7edc18ad 2026-02-18T06:23:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ trash_move cluster2 mirror test 2026-02-18T06:23:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 trash move mirror/test 2026-02-18T06:23:26.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test deleted 2026-02-18T06:23:26.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:26.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:23:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:23:26.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:23:26.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:23:26.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:23:26.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:23:27.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted '' 2026-02-18T06:23:27.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:27.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:27.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:27.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:23:27.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:23:27.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:23:27.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:23:27.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ trash_restore cluster2 mirror test 112f7edc18ad journal 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=112f7edc18ad 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:23:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 trash restore mirror/112f7edc18ad 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:27.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:27.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:28.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:28.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:28.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:28.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:28.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:28.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:28.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:28.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:28.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:23:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:23:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:28.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:28.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:23:28.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:23:28.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:28.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:23:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:23:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:28.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:28.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:28.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:28.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:23:28.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:28.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:28.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:28.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:23:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:23:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:28.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:28.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:28.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:28.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:23:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:23:29.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:29.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:29.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:29.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:29.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:29.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:29.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:29.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:29.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:29.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:23:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:23:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:23:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:23:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:23:29.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:29.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:29.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:23:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:23:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:23:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:23:31.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:31.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:31.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:31.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:31.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:31.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:31.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:31.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:31.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:31.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:31.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:31.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:31.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:31.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:31.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed (with rbd-mirror on one cluster)' 2026-02-18T06:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:23:31.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:23:31 TEST: check if removed 'images'\''' OMAP are removed '(with' rbd-mirror on one 'cluster)' 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:23:31 TEST: check if removed images' OMAP are removed (with rbd-mirror on one cluster) 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:31.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test 2026-02-18T06:23:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2026-02-18T06:23:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test 2026-02-18T06:23:31.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:23:31.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_in_omap cluster1 mirror 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=status_global 2026-02-18T06:23:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:31.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:31.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:31.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2026-02-18T06:23:31.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:31.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:31.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:31.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=status_global 2026-02-18T06:23:31.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-18T06:23:31.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:31.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c status_global 2026-02-18T06:23:31.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:31.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:31.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:23:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 0 2026-02-18T06:23:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=1 2026-02-18T06:23:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 -eq 0 ']' 2026-02-18T06:23:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:31.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:32.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=status_global 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:32.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:23:32.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c status_global 2026-02-18T06:23:32.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:23:32.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:23:32.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:32.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:23:32.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_ 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:32.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_ 2026-02-18T06:23:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-18T06:23:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:23:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_ 2026-02-18T06:23:32.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:23:32.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:23:32.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirror_leader 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_map 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:23:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:32.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:23:32.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 0 2026-02-18T06:23:32.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=1 2026-02-18T06:23:32.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:32.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 -eq 0 ']' 2026-02-18T06:23:32.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:32.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:33.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:23:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:23:33.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-18T06:23:33.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:33.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:23:33.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:33.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:33.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:23:33.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 0 2026-02-18T06:23:33.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=1 2026-02-18T06:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 -eq 0 ']' 2026-02-18T06:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:23:35.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:35.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:35.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:23:35.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:23:35.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:23:35.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:23:35.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:35.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:23:35.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:35.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_in_omap cluster2 mirror 2026-02-18T06:23:35.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:35.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:35.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2026-02-18T06:23:35.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:35.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:35.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:23:35.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=status_global 2026-02-18T06:23:35.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:35.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:35.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:35.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2026-02-18T06:23:35.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:35.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:35.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:35.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=status_global 2026-02-18T06:23:35.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-18T06:23:35.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:35.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:35.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:35.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:23:35.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c status_global 2026-02-18T06:23:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:23:36.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_ 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_ 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:36.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_ 2026-02-18T06:23:36.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:23:36.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:23:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:23:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:23:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:23:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2026-02-18T06:23:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirror_leader 2026-02-18T06:23:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_map 2026-02-18T06:23:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:23:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:23:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:23:36.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error getting omap key set mirror/rbd_mirror_leader: (2) No such file or directory 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:36.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:23:36.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:36.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2026-02-18T06:23:36.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:36.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:36.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:36.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:36.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:36.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:37.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:37.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:37.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:37.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:37.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:37.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:37.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:37.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:37.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:37.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:37.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:37.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:37.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:37.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:37.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:37.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:37.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:37.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:37.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:23:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:23:37.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:23:37.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:37.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:37.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:37.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:23:37.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:37.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:23:37.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:37.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:37.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:23:37.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:23:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:37.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:37.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:37.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:37.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:23:37.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:37.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:37.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:37.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:37.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:23:38.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:38.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:38.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:38.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:38.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:38.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:38.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:38.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:38.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:38.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:23:38.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:38.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:38.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:23:38.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:23:38.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:38.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:38.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:38.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:23:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:23:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:23:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:38.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:38.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:23:38.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:23:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:38.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:38.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:23:38.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:23:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:23:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:23:38.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:38.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:23:40.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:40.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:40.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:40.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:40.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:40.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:40.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:40.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:40.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:40.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:40.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:40.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:40.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:40.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:40.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:40.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:40.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test 100 2026-02-18T06:23:40.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:40.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:40.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:40.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:23:40.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:23:40.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:40.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:23:40.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:23:40.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:23:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:23:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.44 bytes/sec: 30 MiB/s 2026-02-18T06:23:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:23:40.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:40.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:23:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:23:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:23:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:23:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:23:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:23:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:23:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:40.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:40.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:23:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:23:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:23:40.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:40.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:40.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:40.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:23:40.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:23:40.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:23:40.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:40.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:40.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:23:40.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:23:41.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:23:41.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:23:41.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:23:41.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:23:41.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:23:41.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:23:41.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:23:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:23:41.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:23:41.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:23:41.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:23:41.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:23:41.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:23:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:23:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:23:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' '' 2026-02-18T06:23:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:23:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = '' 2026-02-18T06:23:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:23:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:23:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:23:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:23:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:23:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:23:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:23:41.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:23:41.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:23:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:23:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:23:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:23:41.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:23:41.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:23:41.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:23:41.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:23:41.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=1 2026-02-18T06:23:41.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 1 -a 101 -le 1 2026-02-18T06:23:41.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:23:41.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:23:41.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:23:41.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:23:41.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:41.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:41.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:41.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:41.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:41.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:41.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:41.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:23:41.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:23:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:23:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:23:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:23:41.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:23:41.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:23:41.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:23:41.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:23:41.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:23:41.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:23:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:23:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:23:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:23:41.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:23:41.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:23:41.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:23:41.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:23:41.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:23:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:23:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:42.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:42.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:39' 2026-02-18T06:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:42.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:23:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:42.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:42.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:42.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:42.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:42.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:42.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:42.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:39' 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:23:39' 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:39' 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:42.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:42.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}43444348mirror.0trial1302026-02-18 06:23:39' 2026-02-18T06:23:42.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:42.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:42.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:39' 2026-02-18T06:23:42.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:42.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:42.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:42.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:42.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:42.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:42.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:42.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: failover and failback' 2026-02-18T06:23:42.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:23:42.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:23:42.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:23:42 TEST: failover and failback 2026-02-18T06:23:42.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:23:42 TEST: failover and failback 2026-02-18T06:23:42.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster2 2026-02-18T06:23:42.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:42.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:0 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:23:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:23:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:23:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:23:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:1 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:23:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:2 2026-02-18T06:23:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:23:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:23:42.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:42.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:23:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:3 2026-02-18T06:23:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:23:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:23:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:23:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:23:42.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:42.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:42.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:42.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:23:42.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:42.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:42.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:23:42.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:42.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:42.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:23:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:43.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:43.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:43.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:43.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:43.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:43.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:43.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:43.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:43.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:43.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:43.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:43.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:43.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:43.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:23:43.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:43.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:45.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:23:45.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:23:45.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:45.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:46.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:44 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:44' 2026-02-18T06:23:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:23:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:23:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:23:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:23:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:48.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:48.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:47 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:47' 2026-02-18T06:23:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:23:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:23:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:23:48.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:23:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:23:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:48.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:48.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:47' 2026-02-18T06:23:48.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:23:47' 2026-02-18T06:23:48.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:48.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:48.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:48.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:48.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:47' 2026-02-18T06:23:48.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:48.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:48.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:23:47cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:23:47' 2026-02-18T06:23:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:47' 2026-02-18T06:23:48.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:23:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:23:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:23:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:23:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:23:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:49.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:23:49.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:49.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:47 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:23:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:23:49.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:23:49.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:23:49.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:47' 2026-02-18T06:23:49.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:23:49.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:23:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:23:49.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:49.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:23:49.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:49.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:23:49.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:49.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:49.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:49.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:47' 2026-02-18T06:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:23:47' 2026-02-18T06:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:49.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:47' 2026-02-18T06:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:49.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:23:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:23:47' 2026-02-18T06:23:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:47' 2026-02-18T06:23:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:49.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:23:49.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:23:49.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:23:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:23:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:23:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:23:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:23:49.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:50.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:50.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:50.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:50.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:50.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:50.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:50.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:50.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:51.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:23:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:23:51.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:23:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:51.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:23:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:23:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:23:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:53.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:53.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:53.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:53.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:53.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:53.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:53.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:53.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:53.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:53.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:53.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:23:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test 100 2026-02-18T06:23:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:23:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:23:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:23:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:23:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:23:53.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:23:53.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:23:53.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.47 bytes/sec: 33 MiB/s 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:23:53.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:23:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:23:53.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:23:53.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:53.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:53.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:53.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:23:53.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:53.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:23:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:23:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:23:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:53.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:53.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:53.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:53.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:23:53.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:53.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:23:53.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:23:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:53.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:53.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:23:53.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:23:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:23:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:23:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:23:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:23:53.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2026-02-18T06:23:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:23:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:23:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:23:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:23:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:23:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:23:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:23:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:23:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:23:53.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:53.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:23:53.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:23:53.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:23:53.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:23:53.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:23:53.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:23:53.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=6, entry_tid=101' 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=1, tag_tid=6, entry_tid=101' 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:53.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:54.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:54.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:23:54.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:54.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:23:54.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:23:54.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:23:54.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:54.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:52 2026-02-18T06:23:54.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:23:54.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:23:54.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:54.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2026-02-18T06:23:54.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:52' 2026-02-18T06:23:54.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:23:54.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:23:54.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:23:54.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:23:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:54.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:54.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:54.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:54.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:52' 2026-02-18T06:23:54.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:23:52' 2026-02-18T06:23:54.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:54.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:54.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:54.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:23:54.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:52' 2026-02-18T06:23:54.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:54.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:54.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:23:52cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}2026-02-18 06:23:52' 2026-02-18T06:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:54.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:52' 2026-02-18T06:23:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:23:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:23:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:23:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:23:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:23:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:23:55.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:23:55.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:23:55.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:23:55.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:52 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:52' 2026-02-18T06:23:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:23:55.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:23:55.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:23:55.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2026-02-18T06:23:55.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:23:55.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:55.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:23:55.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:55.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:23:55.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:23:55.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:23:55.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:23:55.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:52' 2026-02-18T06:23:55.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:23:52' 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:52' 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:23:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:23:55.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:23:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:23:52' 2026-02-18T06:23:55.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:23:55.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:23:55.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:52' 2026-02-18T06:23:55.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:23:55.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:23:55.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:23:55.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:23:55.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:23:55.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:23:55.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:23:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:23:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:23:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:23:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:23:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:23:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:23:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:23:55.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:23:55.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:23:55.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:23:55.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:23:55.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:23:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:23:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:23:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:23:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:23:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:23:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:23:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:55.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:23:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:23:56.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:56.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:56.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:56.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:56.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:56.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:56.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:56.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:56.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:56.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:56.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:56.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:56.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:56.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:56.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:56.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:56.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:56.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:56.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:56.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:56.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:23:56.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:23:56.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:23:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:23:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:23:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:23:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:23:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:23:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:23:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:23:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:23:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:23:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:23:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:23:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:23:58.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:23:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:23:59.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:59.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:59.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:23:59.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:23:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:23:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:23:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:00.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:24:00.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:00.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:57 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:57' 2026-02-18T06:24:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:00.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:00.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:57' 2026-02-18T06:24:00.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:23:57' 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:57' 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:23:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:23:57' 2026-02-18T06:24:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:57' 2026-02-18T06:24:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:00.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:00.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:00.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:00.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:24:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:01.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:01.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:01.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:57 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:23:57' 2026-02-18T06:24:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:01.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:57' 2026-02-18T06:24:01.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:23:57' 2026-02-18T06:24:01.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:23:57' 2026-02-18T06:24:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:23:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:23:57' 2026-02-18T06:24:01.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:01.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:01.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:23:57' 2026-02-18T06:24:01.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:24:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:24:01.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:01.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:02.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:02.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:02.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:02.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:02.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:02.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:02.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:02.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:02.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:02.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:02.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:02.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:02.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:02.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:02.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:02.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:02.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:02.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:02.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:02.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:02.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:02.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:24:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:02.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:24:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:04.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:04.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:04.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:04.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:04.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:04.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:04.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:04.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:04.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:04.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:04.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:24:04.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:24:04.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:24:04.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:04.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:05.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:05.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:05.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:05.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:05.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:05.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:05.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:05.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:05.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:05.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:05.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:05.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:05.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:05.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:05.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:05.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:05.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:05.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:05.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:05.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:05.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:05.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:05.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:05.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:24:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:07.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:07.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:07.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:07.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:07.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:07.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:07.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:07.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:07.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:07.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:07.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:07.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:07.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:08.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:07 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:07' 2026-02-18T06:24:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:08.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:07' 2026-02-18T06:24:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:24:07' 2026-02-18T06:24:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:07' 2026-02-18T06:24:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:08.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:24:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:24:07' 2026-02-18T06:24:08.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:08.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:08.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:07' 2026-02-18T06:24:08.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:08.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:08.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:08.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:08.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:24:08.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:08.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:08.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:08.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:08.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:08.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:08.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:08.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:09.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:07 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:07' 2026-02-18T06:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:09.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:09.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:09.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:09.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:09.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:07' 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:24:07' 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:07' 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:09.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:24:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:24:07' 2026-02-18T06:24:09.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:09.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:09.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:07' 2026-02-18T06:24:09.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:09.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:09.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:09.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:09.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:09.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:09.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:09.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:24:09.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:09.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:24:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:09.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:09.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:10.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:10.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:10.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:10.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:10.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:24:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:12.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:24:12.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:12.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:12.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:12.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:12.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:12.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:12.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:12.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:12.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:12.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:12.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:12.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:12.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:12.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:12.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:12.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:12.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:12.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:12.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:24:12.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:12.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:24:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:24:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:24:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:12.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:24:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:24:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:24:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:24:13.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:13.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:24:13.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:24:13.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:13.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:13.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:13.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:13.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:13.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:13.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:13.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:24:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:13.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:13.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:13.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=8, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=11, entry_tid=0' 2026-02-18T06:24:13.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:24:13.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:24:13.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:13.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:13.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=8, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=8, entry_tid=0' 2026-02-18T06:24:13.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=11, entry_tid=0' -a 'object_number=0, tag_tid=11, entry_tid=0' = 'object_number=0, tag_tid=8, entry_tid=0' 2026-02-18T06:24:13.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=8, entry_tid=0' '!=' '' 2026-02-18T06:24:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:24:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=8, entry_tid=0' = '' 2026-02-18T06:24:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=0, tag_tid=8, entry_tid=0' 2026-02-18T06:24:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=11, entry_tid=0' 2026-02-18T06:24:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:24:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=11 2026-02-18T06:24:13.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=8, entry_tid=0' 2026-02-18T06:24:13.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:24:13.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:13.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=8 2026-02-18T06:24:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=11, entry_tid=0' 2026-02-18T06:24:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:24:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=0 2026-02-18T06:24:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=8, entry_tid=0' 2026-02-18T06:24:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:24:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=0 2026-02-18T06:24:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 11 = 8 -a 0 -le 0 2026-02-18T06:24:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:24:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:24:13.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:24:13.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:13.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:13.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:24:13.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:13.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:13.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:13.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:13.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:13.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:13.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:13.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:13.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:13.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:13.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:13.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:13.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:13.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:13.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:13.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:13.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=11, entry_tid=0' 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:13.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:13.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:13.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:13.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:13.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:13.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:13.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:13.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=11, entry_tid=0' 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=11, entry_tid=0' -a 'object_number=0, tag_tid=11, entry_tid=0' = 'object_number=0, tag_tid=11, entry_tid=0' 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:13.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:13.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:24:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:24:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:12 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:14.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:14.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:12' 2026-02-18T06:24:14.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:24:14.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:24:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:24:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2026-02-18T06:24:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:14.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:12' 2026-02-18T06:24:14.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:24:12' 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:12' 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:14.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}43444348mirror.0trial1302026-02-18 06:24:12cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:24:12' 2026-02-18T06:24:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:14.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:14.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:12' 2026-02-18T06:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:14.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:24:14.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:14.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:24:14.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:14.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:14.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:14.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:15.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:24:15.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:15.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:12 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2026-02-18T06:24:15.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:12' 2026-02-18T06:24:15.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:24:15.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:24:15.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:24:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:15.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:15.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:15.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:15.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:15.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:15.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:12' 2026-02-18T06:24:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:24:12' 2026-02-18T06:24:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:12' 2026-02-18T06:24:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:24:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}2026-02-18 06:24:12' 2026-02-18T06:24:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:12' 2026-02-18T06:24:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:24:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:24:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:15.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:24:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:24:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:15.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:24:15.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:24:15.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:24:15.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:15.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:15.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:15.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:24:15.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:24:15.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:24:15.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:15.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:15.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:24:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:24:15.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:24:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:16.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:17.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:17.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:17.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:17.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:17.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:17.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:17.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:17.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:17.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:17.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:17.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:17.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:24:17.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:17.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:19.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:24:19.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:19.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:19.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:19.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:19.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:19.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:19.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:19.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:20.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:20.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:17 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:20.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:20.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:17' 2026-02-18T06:24:20.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:20.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:20.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:20.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:20.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:20.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:20.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:20.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:20.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:20.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:20.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:20.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:17' 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:24:17' 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:17' 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:20.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:24:17cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:24:17' 2026-02-18T06:24:20.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:20.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:20.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:17' 2026-02-18T06:24:20.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:20.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:20.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:20.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:20.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:20.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:20.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:20.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:24:20.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:20.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:21.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:21.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:17 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:17' 2026-02-18T06:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:21.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:21.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:21.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:21.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:17' 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:24:17' 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:17' 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:24:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:24:17' 2026-02-18T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:17' 2026-02-18T06:24:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:21.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:21.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:21.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:21.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:24:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:24:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:24:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:24:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:24:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:22.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:22.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:22.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:22.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:22.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:22.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:22.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:22.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:24:22.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:22.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:24:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:24.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:24.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:24.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:24.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:24.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:24.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:24.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster1 mirror test 100 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:24:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:24:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:24:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:24:24.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:24:24.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:24:24.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.44 bytes/sec: 30 MiB/s 2026-02-18T06:24:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:24:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:24:24.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:24:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:24:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:24:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:24:24.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:24:24.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:24.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:24.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:24.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:24.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:24.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:24.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:24.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:24:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:24:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:24.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:24:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:24.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:24.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:24.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:24.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:24.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2026-02-18T06:24:24.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:24.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2026-02-18T06:24:24.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:24:24.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:24.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:24.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:24.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:24.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2026-02-18T06:24:24.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:24.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos= 2026-02-18T06:24:24.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = '' 2026-02-18T06:24:24.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test '' '!=' '' 2026-02-18T06:24:24.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:24.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:24:25.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:24:25.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:25.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:25.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:25.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:25.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:25.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:25.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:25.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:25.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:25.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:25.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:25.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:25.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:25.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2026-02-18T06:24:25.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:25.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2026-02-18T06:24:25.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:24:25.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:25.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:25.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:25.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:24:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:25.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:25.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:25.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:25.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:25.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2026-02-18T06:24:25.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:25.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos= 2026-02-18T06:24:25.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = '' 2026-02-18T06:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test '' '!=' '' 2026-02-18T06:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:25.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.8 2026-02-18T06:24:26.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:24:26.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:26.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:26.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:26.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:26.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:26.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:26.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:24:26.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:24:26.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:26.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:26.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:24:26.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:26.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:26.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:26.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:26.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:26.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:26.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:26.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:26.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:26.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:26.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:26.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:26.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:26.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2026-02-18T06:24:26.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:24:26.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:26.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:26.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:26.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:24:26.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:26.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:26.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:26.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:26.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:26.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:26.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:26.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:26.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:26.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:26.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:26.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:26.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:26.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:26.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:26.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=13, entry_tid=101' 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = 'object_number=1, tag_tid=13, entry_tid=101' 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:27.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:24:27.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:27.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:27.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:27.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:27.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:27.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:27.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:27.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:27.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:27.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:27.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:27.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:27.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:27.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:27.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:22 2026-02-18T06:24:27.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2026-02-18T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:22' 2026-02-18T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:24:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:27.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:27.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:27.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:22' 2026-02-18T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:24:22' 2026-02-18T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:22' 2026-02-18T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:27.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:24:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}2026-02-18 06:24:22' 2026-02-18T06:24:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:22' 2026-02-18T06:24:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:24:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:24:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:27.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:27.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster2 mirror test 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying primary_position 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:27.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:24:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:28.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying primary_position '' 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:28.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:28.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:28.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:22 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:27' 2026-02-18T06:24:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:24:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:24:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:24:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2026-02-18T06:24:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:28.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:28.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:28.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:22' 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:24:22' 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:22' 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:28.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:28.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}45714576mirror.0trial1302026-02-18 06:24:22cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:24:27' 2026-02-18T06:24:28.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:28.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:28.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:22' 2026-02-18T06:24:28.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:28.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:24:28.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:28.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:24:28.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:28.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:28.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:28.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:24:28.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:24:28.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:28.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:24:28.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:24:28.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:24:28.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:28.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:28.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:28.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:24:28.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:28.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:24:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:28.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:28.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:24:28.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:24:28.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:24:28.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:28.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:28.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:28.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:29.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:29.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:29.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:29.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:29.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:29.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:29.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:29.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:29.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:29.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:29.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:29.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:29.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:29.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:29.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:29.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:29.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:29.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:29.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:29.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:29.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:29.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:24:29.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:29.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:31.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:31.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:31.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:31.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:31.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:31.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:31.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:31.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:31.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:31.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:31.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:31.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:31.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:31.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:31.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:31.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:31.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:31.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:31.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:31.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:24:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:24:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:31.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:31.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:31.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:31.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:31.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:32.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:24:32.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:32.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:32.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:32.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:32.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:32.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:32 2026-02-18T06:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:32.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:32.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:32.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:32.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:32' 2026-02-18T06:24:32.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:32.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:32.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:32.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:32.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:32' 2026-02-18T06:24:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:24:32' 2026-02-18T06:24:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:32.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:32.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:32.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:32' 2026-02-18T06:24:32.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:32.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:32.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:32.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:24:32cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:24:32' 2026-02-18T06:24:32.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:32.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:32.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:32' 2026-02-18T06:24:32.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:32.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:32.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:32.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:32.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:32.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:32.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:32.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:32.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:24:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:33.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:34.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:34.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:34.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:34.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:34.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:34.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:32 2026-02-18T06:24:34.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:32' 2026-02-18T06:24:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:34.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:34.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:34.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:34.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:34.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:34.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:34.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:32' 2026-02-18T06:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:24:32' 2026-02-18T06:24:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:32' 2026-02-18T06:24:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:34.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:24:32cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:24:32' 2026-02-18T06:24:34.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:34.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:32' 2026-02-18T06:24:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:34.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:34.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:34.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:34.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:34.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:24:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:24:34.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:24:34.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:34.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:24:34.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:34.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:34.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:35.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:24:35.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:35.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:35.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:35.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:35.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:35.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:35.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:35.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:35.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:35.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:35.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:35.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:35.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:35.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:35.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:35.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:35.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:35.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:35.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:35.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:35.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:24:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:24:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:37.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:37.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:37.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:37.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:37.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:37.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:37.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:37.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:24:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:24:41.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:24:41.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:41.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:41.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:41.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:41.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:41.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:41.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:41.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:41.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:41.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:41.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:41.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:41.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:41.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:41.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:41.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:41.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test 100 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:24:41.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:24:41.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:24:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:24:41.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.44 bytes/sec: 30 MiB/s 2026-02-18T06:24:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:24:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:24:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:41.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:24:41.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:24:41.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:24:41.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:41.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:41.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:24:41.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:41.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:41.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:41.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:41.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:41.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:41.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:41.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:41.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:41.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:41.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:24:41.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:41.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:41.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:41.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:41.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:41.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:41.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' '' 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' = '' 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:41.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:24:41.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:41.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=18 2026-02-18T06:24:41.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:24:41.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:41.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=17 2026-02-18T06:24:41.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:41.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:24:41.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:41.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:24:41.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:24:41.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:24:41.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=0 2026-02-18T06:24:41.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 18 = 17 -a 101 -le 0 2026-02-18T06:24:41.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:24:41.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:41.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:24:41.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:24:41.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:41.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:41.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:24:41.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:41.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:41.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:41.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:24:41.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:41.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:41.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:41.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:24:41.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:41.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:41.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:41.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:41.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:41.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:24:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:41.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:41.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:41.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:41.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' 'object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:24:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:42.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:42.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:24:42.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:42.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:42.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:42.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:42.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:42.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:42.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:42.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:42.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:42.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:42.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:42.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:42.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:42.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:42.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2026-02-18T06:24:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:42.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' 'object_number=0, tag_tid=17, entry_tid=0' 2026-02-18T06:24:42.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:42.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.8 2026-02-18T06:24:43.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:24:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:43.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:43.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:24:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:43.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:43.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:43.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:43.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:24:43.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:43.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:43.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:43.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:43.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:43.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:43.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:43.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:43.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:43.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:43.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:43.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:43.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:43.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:24:43.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:43.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:43.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:43.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:24:43.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:43.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:43.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:24:43.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=1, tag_tid=18, entry_tid=101' 2026-02-18T06:24:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:24:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:43.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:24:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:24:44.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:44.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:44.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:44.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2026-02-18T06:24:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:44.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:37 2026-02-18T06:24:44.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:44.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:44.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:44.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:44.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:42' 2026-02-18T06:24:44.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:24:44.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:24:44.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:24:44.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2026-02-18T06:24:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:37' 2026-02-18T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:24:37' 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:37' 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}43444348mirror.0trial1302026-02-18 06:24:37cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:24:42' 2026-02-18T06:24:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:44.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:37' 2026-02-18T06:24:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:24:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:24:44.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:44.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:44.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:44.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:24:44.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:44.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:44.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:44.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:45.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:45.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:45.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:45.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:45.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:45.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:45.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:42 2026-02-18T06:24:45.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:45.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:45.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:45.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2026-02-18T06:24:45.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:37' 2026-02-18T06:24:45.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:24:45.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:24:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:24:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:45.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:45.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:45.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:45.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:42' 2026-02-18T06:24:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:24:42' 2026-02-18T06:24:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:42' 2026-02-18T06:24:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:24:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}2026-02-18 06:24:37' 2026-02-18T06:24:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:45.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:42' 2026-02-18T06:24:45.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:45.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:24:45.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:45.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:24:45.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:24:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:24:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:24:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:45.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:24:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:24:45.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:24:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:24:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:24:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:45.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:45.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:45.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:24:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:24:45.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:45.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:24:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:24:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: failover / failback loop' 2026-02-18T06:24:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:24:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:24:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:24:45 TEST: failover / failback loop 2026-02-18T06:24:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:24:45 TEST: failover / failback loop 2026-02-18T06:24:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 1 20 2026-02-18T06:24:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:24:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:24:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:24:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:24:45.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:24:45.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:45.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:45.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:45.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:24:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:24:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:24:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:46.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:46.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:46.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:46.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:46.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:46.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:46.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:46.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:46.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:46.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:46.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:46.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:46.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:46.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:46.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:46.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:46.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:46.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:46.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:46.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:46.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:46.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:24:46.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:46.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:48.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:24:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:48.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:48.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:24:48.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:24:48.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:24:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:24:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:24:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:24:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:24:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:24:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:24:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:24:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:24:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:24:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:24:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:24:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:24:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:48.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:24:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:24:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:24:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:24:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:48.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:24:48.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:49.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:49.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:49.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:49.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:24:49.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:49.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:24:49.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:49.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:50.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:24:50.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:50.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:50.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:50.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:50.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:47 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:47' 2026-02-18T06:24:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:50.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:50.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:50.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:50.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:50.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:50.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:50.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:50.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:47' 2026-02-18T06:24:50.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:24:47' 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:47' 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:50.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:24:47cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:24:47' 2026-02-18T06:24:50.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:50.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:50.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:47' 2026-02-18T06:24:50.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:50.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:50.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:50.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:50.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:50.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:50.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:51.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:51.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:47 2026-02-18T06:24:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:24:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:24:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:47' 2026-02-18T06:24:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:24:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:24:51.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:51.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:24:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:47' 2026-02-18T06:24:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:24:47' 2026-02-18T06:24:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:24:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:47' 2026-02-18T06:24:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:51.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:24:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:24:47' 2026-02-18T06:24:51.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:51.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:51.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:47' 2026-02-18T06:24:51.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:51.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:24:51.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:24:51.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:24:51.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:24:51.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:51.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:51.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:24:51.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:51.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:24:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:24:51.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:24:51.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:51.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:51.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:52.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:52.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:52.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:52.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:52.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:52.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:24:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:24:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:24:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:24:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:54.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:54.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:54.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:54.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:54.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:54.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:54.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:54.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:54.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:24:54.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:54.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:54.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:24:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:24:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:24:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:24:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:24:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:24:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:24:54.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:24:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:24:54.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:54.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:24:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:54.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:54.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:24:54.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:54.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:24:54.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:54.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:54.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:54.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:54.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:54.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:54.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:54.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:54.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:54.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:54.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:54.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:54.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=19, entry_tid=0' 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:24:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:24:54.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:24:54.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:24:54.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:54.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:24:54.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:24:54.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:24:54.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:54.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:24:54.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:24:54.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=19, entry_tid=0' 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=19, entry_tid=0' -a 'object_number=0, tag_tid=19, entry_tid=0' = 'object_number=0, tag_tid=19, entry_tid=0' 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:54.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:54.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:54.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:55.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:55.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:24:55.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:55.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:55.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:55.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:55.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:55.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:52 2026-02-18T06:24:55.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:55.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:24:55.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:55.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2026-02-18T06:24:55.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:52' 2026-02-18T06:24:55.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:24:55.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:24:55.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:55.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:24:55.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:55.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:55.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:55.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:55.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:55.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:55.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:55.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:55.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:52' 2026-02-18T06:24:55.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:24:52' 2026-02-18T06:24:55.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:24:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:52' 2026-02-18T06:24:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:55.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:24:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}2026-02-18 06:24:52' 2026-02-18T06:24:55.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:55.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:55.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:52' 2026-02-18T06:24:55.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:55.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:24:55.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:24:55.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:24:55.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:24:55.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:55.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:55.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:56.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:24:56.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:56.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:24:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:24:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:24:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:24:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2026-02-18T06:24:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:24:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:52 2026-02-18T06:24:57.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:24:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:24:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:24:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:24:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:52' 2026-02-18T06:24:57.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:24:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:24:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:24:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2026-02-18T06:24:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:24:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:57.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:24:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:24:57.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:24:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:24:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:24:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:52' 2026-02-18T06:24:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:24:52' 2026-02-18T06:24:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:24:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:52' 2026-02-18T06:24:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:24:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:24:57.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:24:57.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}45714576mirror.0trial1302026-02-18 06:24:52cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:24:52' 2026-02-18T06:24:57.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:24:57.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:24:57.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:52' 2026-02-18T06:24:57.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:24:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:24:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:24:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:24:57.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:24:57.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:24:57.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:24:57.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:57.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:57.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:24:58.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:24:58.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:24:58.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:24:58.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:24:58.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:24:58.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:24:58.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:24:58.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:24:58.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:24:58.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:24:58.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:58.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:24:58.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:24:58.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:24:58.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:58.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:24:58.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:24:58.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:24:58.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:24:58.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:24:58.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:58.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:24:58.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:24:58.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:24:58.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:24:58.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:24:58.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:24:58.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:24:58.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:24:58.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:24:58.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:24:58.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:24:58.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:24:58.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:24:58.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:24:58.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:24:58.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:00.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:00.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:00.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:00.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:00.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:00.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:00.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:00.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:00.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:00.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:00.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:00.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:00.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:00.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:00.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:00.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:00.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:00.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:00.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:25:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:25:00.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:25:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:57 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:01.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:25:01.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:58' 2026-02-18T06:25:01.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:01.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:01.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:01.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:01.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:01.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:01.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:01.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:01.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:01.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:57' 2026-02-18T06:25:01.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:24:57' 2026-02-18T06:25:01.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:01.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:24:57' 2026-02-18T06:25:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:24:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedstopped2026-02-18 06:24:58' 2026-02-18T06:25:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:01.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:01.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:24:57' 2026-02-18T06:25:01.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:01.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:01.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:02.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:25:02.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:02.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:02.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:02.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:02.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:02.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:25:02.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:02.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:58 2026-02-18T06:25:02.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:02.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:02.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:02.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:02.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:24:57' 2026-02-18T06:25:02.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:02.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:25:02.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:02.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:04.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:04.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:04.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:04.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:04.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:04.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:04.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:04.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:02 2026-02-18T06:25:04.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:04.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:04.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:04.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:04.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:02' 2026-02-18T06:25:04.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:04.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:04.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:04.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:04.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:04.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:04.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:04.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:04.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:04.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:04.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:04.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:02' 2026-02-18T06:25:04.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:25:02' 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:02' 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:04.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:04.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:25:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:25:02' 2026-02-18T06:25:04.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:04.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:04.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:02' 2026-02-18T06:25:04.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:04.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:04.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:04.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:25:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:25:04.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:25:04.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:25:04.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:05.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:25:05.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:05.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:05.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:05.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:05.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:05.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:05.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:05.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:05.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:05.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:05.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:05.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:05.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:05.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:05.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:05.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:05.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:05.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:05.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:05.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:05.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:05.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:05.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:05.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:05.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:25:05.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:05.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:25:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:07.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:07.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:07.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:07.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:07.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:07.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:07.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:07.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:07.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:07.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:07.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:07.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:07.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:07.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:07.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:07.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:07.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:25:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:07.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:07.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:25:07.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:25:07.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:25:07.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:25:07.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:25:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:25:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:25:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:25:07.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:07.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:25:07.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:07.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:08.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:08.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:25:08.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:25:08.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:08.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:08.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:08.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:08.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:08.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:08.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:25:08.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:25:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:25:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:25:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:25:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=20, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=23, entry_tid=0' 2026-02-18T06:25:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:25:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:25:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:25:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:08.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:25:08.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:08.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:08.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:08.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=20, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=20, entry_tid=0' 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=23, entry_tid=0' -a 'object_number=0, tag_tid=23, entry_tid=0' = 'object_number=0, tag_tid=20, entry_tid=0' 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=20, entry_tid=0' '!=' '' 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=20, entry_tid=0' = '' 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=0, tag_tid=20, entry_tid=0' 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=23, entry_tid=0' 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:25:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:25:08.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=23 2026-02-18T06:25:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=20, entry_tid=0' 2026-02-18T06:25:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:25:08.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:25:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=20 2026-02-18T06:25:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=23, entry_tid=0' 2026-02-18T06:25:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:25:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:25:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=0 2026-02-18T06:25:08.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=20, entry_tid=0' 2026-02-18T06:25:08.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:25:08.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:25:08.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=0 2026-02-18T06:25:08.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 23 = 20 -a 0 -le 0 2026-02-18T06:25:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:25:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:25:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:25:08.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:25:08.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:25:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:25:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:25:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:25:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:25:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:08.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:08.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:08.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:08.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:08.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:25:08.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:08.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:25:08.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:25:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:25:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=23, entry_tid=0' 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:25:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:25:08.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:08.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:08.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=23, entry_tid=0' 2026-02-18T06:25:08.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=23, entry_tid=0' -a 'object_number=0, tag_tid=23, entry_tid=0' = 'object_number=0, tag_tid=23, entry_tid=0' 2026-02-18T06:25:08.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:07 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2026-02-18T06:25:09.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:07' 2026-02-18T06:25:09.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:25:09.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:25:09.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:09.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:25:09.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:09.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:09.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:09.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:09.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:07' 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:25:07' 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:07' 2026-02-18T06:25:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:25:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}2026-02-18 06:25:07' 2026-02-18T06:25:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:07' 2026-02-18T06:25:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:09.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:25:09.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:09.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:25:09.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:09.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:09.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:09.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:25:09.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:09.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:09.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:09.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:09.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:09.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:09.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:09.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:25:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:10.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:10.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:10.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:07 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:07' 2026-02-18T06:25:10.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:25:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:25:10.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:10.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2026-02-18T06:25:10.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:10.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:10.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:10.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:10.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:10.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:10.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:10.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:07' 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:25:07' 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:07' 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}43444348mirror.0trial1302026-02-18 06:25:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:25:07' 2026-02-18T06:25:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:10.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:10.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:07' 2026-02-18T06:25:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:10.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:25:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:25:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:10.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:25:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:25:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:25:10.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:25:10.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:10.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:10.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:11.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:25:11.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:11.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:11.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:11.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:11.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:11.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:11.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:11.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:11.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:11.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:11.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:11.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:11.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:11.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:25:11.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:11.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:13.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:25:13.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:13.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:13.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:13.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:13.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:13.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:13.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:13.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:13.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:13.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:13.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:13.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:13.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:13.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:13.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:13.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:13.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:13.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:13.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:13.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:13.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:13.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:13.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:13.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:25:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:13.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:14.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:12 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:14.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:14.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:12' 2026-02-18T06:25:14.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:14.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:12' 2026-02-18T06:25:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:25:12' 2026-02-18T06:25:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:14.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:12' 2026-02-18T06:25:14.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:14.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:14.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:25:12cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:25:12' 2026-02-18T06:25:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:14.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:12' 2026-02-18T06:25:14.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:15.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:25:15.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:15.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:15.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:15.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:15.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:15.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:15.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:15.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:15.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:15.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:15.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:15.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:15.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:15.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:15.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:12 2026-02-18T06:25:15.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:15.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:15.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:15.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:15.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:12' 2026-02-18T06:25:15.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:15.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:15.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:15.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:15.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:12' 2026-02-18T06:25:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:25:12' 2026-02-18T06:25:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:15.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:12' 2026-02-18T06:25:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:15.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:15.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:25:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:25:12' 2026-02-18T06:25:15.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:15.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:15.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:12' 2026-02-18T06:25:15.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:15.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:15.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:25:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:25:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:25:15.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:25:15.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:25:15.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:15.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:16.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:25:16.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:16.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:16.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:16.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:16.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:16.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:16.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:16.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:16.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:16.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:16.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:16.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:16.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:16.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:17.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:17.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:17.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:17.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:19.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:19.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:19.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:19.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:19.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:19.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:19.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:19.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:19.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:19.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:19.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:19.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:19.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:25:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:25:19.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:25:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:25:19.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:19.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:19.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:19.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:19.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:19.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:25:19.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:19.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:19.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:19.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:25:19.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:19.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=24, entry_tid=0' 2026-02-18T06:25:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:25:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:19.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:19.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:25:19.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=24, entry_tid=0' 2026-02-18T06:25:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=24, entry_tid=0' -a 'object_number=0, tag_tid=24, entry_tid=0' = 'object_number=0, tag_tid=24, entry_tid=0' 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:20.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:20.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:20.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:20.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:20.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:17 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:17' 2026-02-18T06:25:20.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:25:20.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:25:20.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:20.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:25:20.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:20.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:20.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:20.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:20.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:20.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:17' 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:25:17' 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:17' 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:25:17cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}2026-02-18 06:25:17' 2026-02-18T06:25:20.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:20.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:20.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:17' 2026-02-18T06:25:20.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:25:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:25:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:25:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:20.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:25:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:17 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:17' 2026-02-18T06:25:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:25:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:25:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2026-02-18T06:25:21.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:21.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:21.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:21.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:21.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:21.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:21.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:21.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:17' 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:25:17' 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:17' 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:21.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}45714576mirror.0trial1302026-02-18 06:25:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:25:17' 2026-02-18T06:25:21.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:21.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:17' 2026-02-18T06:25:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:21.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:25:21.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:21.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:25:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:25:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:25:21.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:25:21.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:25:21.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:21.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:21.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:22.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:22.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:22.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:22.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:22.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:22.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:22.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:25:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:24.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:24.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:24.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:24.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:24.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:24.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:24.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:24.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:24.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:25:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:25.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:22 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:23' 2026-02-18T06:25:25.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:25.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:22' 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:25:22' 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:22' 2026-02-18T06:25:25.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:25.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:25.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:25.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:25:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedstopped2026-02-18 06:25:23' 2026-02-18T06:25:25.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:25.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:25.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:22' 2026-02-18T06:25:25.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:25.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:25.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:25.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:25.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:25:25.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:25.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:25.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:25.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:25.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:25.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:25.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:25.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:26.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:26.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:26.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:23 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:26.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:22' 2026-02-18T06:25:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:25:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:26.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:25:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:28.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:28.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:27 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:28.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:27' 2026-02-18T06:25:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:28.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:28.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:28.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:28.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:27' 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:25:27' 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:27' 2026-02-18T06:25:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:28.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:28.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:25:27cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:25:27' 2026-02-18T06:25:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:28.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:27' 2026-02-18T06:25:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:28.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:25:28.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:28.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:28.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:28.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:25:28.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:25:28.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:25:28.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:25:28.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:28.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:28.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:29.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:25:29.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:29.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:29.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:29.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:29.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:29.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:29.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:29.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:29.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:29.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:29.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:29.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:29.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:29.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:29.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:29.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:29.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:29.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:29.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:29.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:29.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:29.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:29.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:30.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:30.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:30.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:30.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:30.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:30.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:25:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:30.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:32.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:25:32.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:32.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:32.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:32.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:32.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:32.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:32.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:32.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:32.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:32.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:32.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:32.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:32.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:32.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:25:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:25:36.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:25:36.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:36.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:36.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:36.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:36.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:36.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:36.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:36.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:36.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:36.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:36.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:36.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:36.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:36.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:36.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:36.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:36.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:36.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:36.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:36.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:36.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:25:36.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:36.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:25:36.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:25:36.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:36.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:25:36.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:25:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:25:36.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:25:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:25:36.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:36.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:36.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:25:36.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:25:36.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:36.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:25:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:36.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:36.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:36.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:36.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:36.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:25:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:36.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:36.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:36.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:36.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:25:36.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:36.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=28, entry_tid=0' 2026-02-18T06:25:36.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:25:36.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:36.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:25:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:25:36.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:36.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=28, entry_tid=0' 2026-02-18T06:25:36.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=28, entry_tid=0' -a 'object_number=0, tag_tid=28, entry_tid=0' = 'object_number=0, tag_tid=28, entry_tid=0' 2026-02-18T06:25:36.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:36.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:36.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:37.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:37.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:37 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2026-02-18T06:25:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:32' 2026-02-18T06:25:37.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:25:37.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:25:37.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:37.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:25:37.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:37.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:37.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:37.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:37.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:37.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:37.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:37.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:37' 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:25:37' 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:37' 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:37.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:37.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:25:37cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}2026-02-18 06:25:32' 2026-02-18T06:25:37.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:37.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:37.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:37' 2026-02-18T06:25:37.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:37.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:25:37.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:37.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:25:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:37.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:37.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:25:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:38.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2026-02-18T06:25:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:32 2026-02-18T06:25:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:37' 2026-02-18T06:25:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:25:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:25:38.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:38.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2026-02-18T06:25:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:32' 2026-02-18T06:25:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:25:32' 2026-02-18T06:25:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:32' 2026-02-18T06:25:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:38.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}43444348mirror.0trial1302026-02-18 06:25:32cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:25:37' 2026-02-18T06:25:38.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:38.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:38.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:32' 2026-02-18T06:25:38.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:38.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:25:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:25:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:25:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:25:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:25:38.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:25:38.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:25:38.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:38.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:38.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:25:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:39.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:39.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:39.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:39.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:39.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:25:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:41.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:25:41.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:41.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:41.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:41.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:41.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:41.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:41.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:41.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:25:41.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:25:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:25:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:41.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:25:41.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:25:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:25:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:41.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:25:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:25:41.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:41.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:41.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:41.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:41.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:25:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:25:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:41.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:41.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:41.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:41.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:41.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:41.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:25:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:42.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:42 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:42' 2026-02-18T06:25:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:42.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:42.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:42.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:42' 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:25:42' 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:42' 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:42.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:25:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:25:42' 2026-02-18T06:25:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:42.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:42' 2026-02-18T06:25:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:42.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:43.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:25:43.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:42 2026-02-18T06:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:42' 2026-02-18T06:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:43.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:43.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:43.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:43.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:43.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:43.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:43.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:42' 2026-02-18T06:25:43.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:25:42' 2026-02-18T06:25:43.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:43.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:42' 2026-02-18T06:25:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:43.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:25:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:25:42' 2026-02-18T06:25:43.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:43.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:43.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:42' 2026-02-18T06:25:43.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:43.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:43.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:43.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:25:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:25:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:25:44.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:25:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:25:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:25:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:25:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:45.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:45.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:25:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:47.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:47.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:47.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:47.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:47.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:47.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:47.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:47.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:47.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:25:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:51.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:51.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:51.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:51.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:51.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:51.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:51.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:51.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:25:51.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:25:51.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:25:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:25:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:25:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:51.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:51.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:51.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:25:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:25:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:51.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:51.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:25:51.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:51.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:51.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:51.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:51.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:51.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:51.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:51.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=29, entry_tid=0' 2026-02-18T06:25:51.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:25:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:25:51.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:51.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:25:51.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:25:51.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:25:51.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:51.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:25:51.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=29, entry_tid=0' 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=29, entry_tid=0' -a 'object_number=0, tag_tid=29, entry_tid=0' = 'object_number=0, tag_tid=29, entry_tid=0' 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:51.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:51.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:51.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:25:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:52.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:52.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:52.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:52.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:52.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:52 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:47' 2026-02-18T06:25:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:25:52.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:25:52.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:52.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:25:52.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:52.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:52.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:52.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:52.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:52.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:52' 2026-02-18T06:25:52.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:25:52' 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:52' 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:52.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:25:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}2026-02-18 06:25:47' 2026-02-18T06:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:52.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:52.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:52' 2026-02-18T06:25:52.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:52.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:25:52.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:25:52.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:25:52.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:25:52.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:52.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:52.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:52.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:53.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:53.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:53.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:53.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:47 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:52' 2026-02-18T06:25:53.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:25:53.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:25:53.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:53.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2026-02-18T06:25:53.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:53.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:53.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:53.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:53.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:47' 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:25:47' 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:47' 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:53.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}45714576mirror.0trial1302026-02-18 06:25:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:25:52' 2026-02-18T06:25:53.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:53.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:47' 2026-02-18T06:25:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:25:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:25:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:25:53.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:25:53.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:53.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:53.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:25:53.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:53.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:53.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:53.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:25:53.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:25:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:25:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:54.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:54.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:54.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:54.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:54.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:54.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:54.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:54.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:54.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:54.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:54.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:54.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:54.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:54.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:54.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:54.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:54.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:25:54.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:25:54.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:54.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:25:54.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:25:54.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:25:54.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:54.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:25:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:25:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:25:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:25:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:25:56.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:25:56.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:25:56.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:25:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:25:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:25:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:25:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:25:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:25:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:25:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:25:56.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:25:56.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:25:56.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:56.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:56.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:25:56.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:25:56.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:25:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:25:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:25:56.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:25:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:25:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:25:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:56.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:56.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:56.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:56.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:56.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:56.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:56.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:57.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:25:57.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:57.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:25:57.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:57 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:57' 2026-02-18T06:25:57.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:57.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:57.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:57.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:57.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:57.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:57.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:57' 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:25:57' 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:57' 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:57.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:57.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:25:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:25:57' 2026-02-18T06:25:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:57.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:57' 2026-02-18T06:25:57.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:57.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:57.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:57.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:57.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:57.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:57.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:57.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:25:57.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:25:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:25:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:25:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:25:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:57 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:25:57' 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:25:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:25:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:25:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:25:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:25:59.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:59.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:25:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:25:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:25:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:57' 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:25:57' 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:25:57' 2026-02-18T06:25:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:25:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:25:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:25:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:25:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:25:57' 2026-02-18T06:25:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:25:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:25:59.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:25:57' 2026-02-18T06:25:59.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:25:59.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:25:59.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:25:59.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:25:59.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:25:59.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:59.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:25:59.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:25:59.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:25:59.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:59.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:59.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:25:59.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:25:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:25:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:00.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:00.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:00.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:00.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:00.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:00.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:00.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:00.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:00.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:00.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:00.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:00.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:00.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:00.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:00.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:00.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:00.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:00.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:00.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:00.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:02.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:02.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:02.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:02.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:02.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:02.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:02.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:02.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:02.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:26:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:06.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:06.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:06.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:06.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:06.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:26:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:26:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:26:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:26:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:26:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:26:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:26:06.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:26:06.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:26:06.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:06.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:26:06.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:06.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:26:06.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:06.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:26:06.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:26:06.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:06.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:26:06.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:26:06.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:06.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:06.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:06.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:06.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:06.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:06.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:26:06.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:06.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:26:06.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:06.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:06.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:06.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:06.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:26:06.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:06.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:06.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=33, entry_tid=0' 2026-02-18T06:26:06.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:26:06.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:06.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:06.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:06.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:06.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:06.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:06.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:06.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:06.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:06.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:06.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=33, entry_tid=0' 2026-02-18T06:26:06.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=33, entry_tid=0' -a 'object_number=0, tag_tid=33, entry_tid=0' = 'object_number=0, tag_tid=33, entry_tid=0' 2026-02-18T06:26:06.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:06.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:26:06.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:06.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:06.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:06.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:06.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:06.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:06.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:06.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:26:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:07.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:07.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:07.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:07.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:07.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:07.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:07 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:02' 2026-02-18T06:26:07.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:26:07.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:26:07.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:07.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:26:07.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:07.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:07.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:07.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:07.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:07.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:07.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:07.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:07' 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:26:07' 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:07' 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:07.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:07.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:26:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}2026-02-18 06:26:02' 2026-02-18T06:26:07.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:07.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:07.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:07' 2026-02-18T06:26:07.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:07.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:26:07.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:07.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:26:07.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:07.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:08.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:08.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:08.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:08.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:02 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:07' 2026-02-18T06:26:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:26:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:26:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:08.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2026-02-18T06:26:08.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:08.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:08.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:08.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:08.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:08.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:02' 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:26:02' 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:02' 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:08.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}43444348mirror.0trial1302026-02-18 06:26:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:26:07' 2026-02-18T06:26:08.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:08.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:08.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:02' 2026-02-18T06:26:08.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:08.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:26:08.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:08.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:26:08.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:08.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:08.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:08.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:26:08.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:26:08.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:08.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:08.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:08.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:26:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:26:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:26:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:08.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:26:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:09.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:09.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:09.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:09.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:09.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:09.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:09.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:09.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:09.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:09.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:09.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:09.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:09.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:09.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:09.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:09.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:09.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:09.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:26:09.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:09.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:11.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:26:11.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:11.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:11.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:11.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:11.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:11.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:11.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:11.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:11.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:11.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:11.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:12.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:26:12.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:12.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:12.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:12.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:12.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:12.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:12.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:12.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:12.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:12 2026-02-18T06:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:12' 2026-02-18T06:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:12' 2026-02-18T06:26:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:26:12' 2026-02-18T06:26:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:12' 2026-02-18T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:13.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:26:12cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:26:12' 2026-02-18T06:26:13.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:13.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:13.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:12' 2026-02-18T06:26:13.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:13.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:13.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:13.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:13.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:13.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:13.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:13.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:12 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:14.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:12' 2026-02-18T06:26:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:14.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:14.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:14.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:14.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:14.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:14.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:14.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:12' 2026-02-18T06:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:26:12' 2026-02-18T06:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:12' 2026-02-18T06:26:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:14.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:26:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:26:12' 2026-02-18T06:26:14.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:14.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:14.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:12' 2026-02-18T06:26:14.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:14.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:14.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:14.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:14.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:26:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:26:14.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:14.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:14.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:26:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:15.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:17.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:26:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:17.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:17.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:17.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:17.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:17.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:17.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:17.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:17.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:26:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:26:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:21.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:21.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:21.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:21.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:21.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:21.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:21.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:21.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:21.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:21.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:21.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:21.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:21.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:21.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:21.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:26:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:21.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:21.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:21.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:21.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:21.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:26:21.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:21.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:21.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:21.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=34, entry_tid=0' 2026-02-18T06:26:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:26:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:26:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:21.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:21.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:26:21.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:21.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:21.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:26:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:21.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:21.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:21.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:21.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:21.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:21.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=34, entry_tid=0' 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=34, entry_tid=0' -a 'object_number=0, tag_tid=34, entry_tid=0' = 'object_number=0, tag_tid=34, entry_tid=0' 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:21.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:21.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:21.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:22.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:22 2026-02-18T06:26:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2026-02-18T06:26:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:17' 2026-02-18T06:26:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:26:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:26:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:26:22.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:22.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:22.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:22.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:22.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:22' 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:26:22' 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:22' 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:26:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}2026-02-18 06:26:17' 2026-02-18T06:26:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:22.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:22' 2026-02-18T06:26:22.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:26:22.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:22.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:26:22.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:22.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:26:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:23.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:17 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:22' 2026-02-18T06:26:23.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:26:23.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:26:23.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:23.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2026-02-18T06:26:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:23.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:23.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:23.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:23.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:23.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:17' 2026-02-18T06:26:23.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:26:17' 2026-02-18T06:26:23.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:23.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:23.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:23.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:23.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:17' 2026-02-18T06:26:23.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:23.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:23.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:23.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}45714576mirror.0trial1302026-02-18 06:26:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:26:22' 2026-02-18T06:26:23.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:23.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:17' 2026-02-18T06:26:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:26:23.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:23.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:26:23.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:26:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:26:23.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:26:23.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:26:23.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:23.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:23.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:23.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:26:23.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:23.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:23.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:23.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:26:23.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:23.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:23.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:24.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:24.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:24.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:24.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:24.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:24.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:24.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:24.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:24.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:24.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:24.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:24.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:26:24.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:26.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:26:26.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:26.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:26.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:26.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:26.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:26.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:26.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:26.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:26.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:26.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:26.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:26.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:26.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:26.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:26.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:26.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:26.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:26.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:26.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:27.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:27.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:28.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:28.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:28.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:27 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:27' 2026-02-18T06:26:28.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:28.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:28.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:28.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:28.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:28.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:28.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:28.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:28.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:28.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:28.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:28.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:27' 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:26:27' 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:27' 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:28.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:26:27cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:26:27' 2026-02-18T06:26:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:28.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:27' 2026-02-18T06:26:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:28.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:29.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:27 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:27' 2026-02-18T06:26:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:29.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:29.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:29.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:27' 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:26:27' 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:27' 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:29.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:29.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:26:27cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:26:27' 2026-02-18T06:26:29.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:29.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:29.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:27' 2026-02-18T06:26:29.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:29.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:26:29.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:26:29.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:30.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:30.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:30.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:30.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:30.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:30.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:30.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:30.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:30.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:30.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:30.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:30.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:30.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:30.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:30.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:30.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:30.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:30.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:30.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:32.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:32.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:32.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:32.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:32.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:32.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:32.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:32.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:32.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:32.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:32.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:32.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:32.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:32.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:32.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:32.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:32.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:32.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:32.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:32.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:32.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:32.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:32.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:32.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:32.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:32.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:32.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:32.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:26:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:36.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:36.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:36.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:36.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:36.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:36.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:36.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:36.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:36.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:36.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:36.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:36.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:36.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:36.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:36.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:26:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:26:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:26:36.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:26:36.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:26:36.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:26:36.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:36.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:26:36.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:36.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:26:36.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:36.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:26:36.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:36.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:26:36.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:36.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:36.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:36.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:26:36.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:36.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:36.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:26:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:36.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:36.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:36.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:36.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:36.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:36.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:36.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=38, entry_tid=0' 2026-02-18T06:26:36.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:36.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:36.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:36.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:36.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:26:36.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=38, entry_tid=0' 2026-02-18T06:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=38, entry_tid=0' -a 'object_number=0, tag_tid=38, entry_tid=0' = 'object_number=0, tag_tid=38, entry_tid=0' 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:37.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:37 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2026-02-18T06:26:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:32' 2026-02-18T06:26:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:26:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:26:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:37.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:26:37.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:37.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:37.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:37' 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:26:37' 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:37' 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:37.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:37.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:26:37cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}2026-02-18 06:26:32' 2026-02-18T06:26:37.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:37.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:37.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:37' 2026-02-18T06:26:37.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:37.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:26:37.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:37.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:26:37.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:38.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:38.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2026-02-18T06:26:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:32 2026-02-18T06:26:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:37' 2026-02-18T06:26:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:26:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:26:38.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:38.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2026-02-18T06:26:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:32' 2026-02-18T06:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:26:32' 2026-02-18T06:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:32' 2026-02-18T06:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}43444348mirror.0trial1302026-02-18 06:26:32cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:26:37' 2026-02-18T06:26:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:38.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:38.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:32' 2026-02-18T06:26:38.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:38.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:26:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:26:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:26:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:26:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:26:38.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:26:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:26:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:38.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:26:38.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:38.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:38.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:38.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:26:38.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:38.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:38.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:26:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:39.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:39.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:39.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:39.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:39.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:39.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:39.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:39.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:39.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:39.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:39.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:39.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:39.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:39.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:26:39.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:39.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:26:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:41.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:41.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:41.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:41.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:41.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:41.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:41.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:41.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:41.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:42.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:42.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:42.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:42.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:42.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:42.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:42.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:42.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:42.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:26:42.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:42.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:43.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:43.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:42 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:42' 2026-02-18T06:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:43.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:43.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:43.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:43.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:43.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:43.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:43.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:43.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:43.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:43.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:43.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:42' 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:26:42' 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:42' 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:26:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:26:42' 2026-02-18T06:26:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:43.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:42' 2026-02-18T06:26:43.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:43.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:43.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:26:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:43.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:43.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:42 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:44.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:42' 2026-02-18T06:26:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:44.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:44.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:44.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:42' 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:26:42' 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:42' 2026-02-18T06:26:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:44.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:26:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:26:42' 2026-02-18T06:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:44.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:42' 2026-02-18T06:26:44.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:44.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:44.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:44.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:26:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:26:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:44.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:45.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:45.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:45.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:45.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:45.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:45.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:45.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:45.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:45.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:45.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:45.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:45.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:47.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:26:47.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:47.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:47.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:47.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:47.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:47.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:47.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:47.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:47.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:47.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:47.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:47.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:47.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:47.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:47.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:47.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:26:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:26:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:26:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:26:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:26:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:26:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:26:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:26:47.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:26:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:26:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:26:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:26:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:26:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:47.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:47.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:47.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:47.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:47.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:47.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:26:48.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:48.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:26:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:48.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:48.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:48.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:48.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:48.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:48.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:48.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=39, entry_tid=0' 2026-02-18T06:26:48.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:26:48.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:48.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:48.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:48.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:48.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:48.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:26:48.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:26:48.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=39, entry_tid=0' 2026-02-18T06:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=39, entry_tid=0' -a 'object_number=0, tag_tid=39, entry_tid=0' = 'object_number=0, tag_tid=39, entry_tid=0' 2026-02-18T06:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:49.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:26:49.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:49.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:47 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:49.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2026-02-18T06:26:49.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:47' 2026-02-18T06:26:49.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:26:49.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:26:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:26:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:49.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:49.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:49.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:49.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:47' 2026-02-18T06:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:26:47' 2026-02-18T06:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:47' 2026-02-18T06:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:49.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:26:47cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}2026-02-18 06:26:47' 2026-02-18T06:26:49.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:49.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:49.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:47' 2026-02-18T06:26:49.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:49.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:26:49.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:26:49.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:26:49.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:26:49.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:49.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:49.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:26:49.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:49.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:49.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:49.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:49.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:26:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:50.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:50.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:50.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:50.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:47 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:26:50.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:26:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:47' 2026-02-18T06:26:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:26:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:26:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:50.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2026-02-18T06:26:50.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:50.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:50.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:50.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:50.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:50.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:50.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:50.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:47' 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:26:47' 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:47' 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:50.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:50.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}45714576mirror.0trial1302026-02-18 06:26:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:26:47' 2026-02-18T06:26:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:47' 2026-02-18T06:26:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:50.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:26:50.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:26:50.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:26:50.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:26:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:26:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:50.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:26:50.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:50.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:51.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:26:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:51.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:51.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:51.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:51.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:51.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:51.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:51.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:51.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:51.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:51.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:51.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:51.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:51.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:26:51.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:51.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:53.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:26:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:26:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:53.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:26:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:26:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:26:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:26:53.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:26:53.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:53.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:53.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:26:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:26:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:53.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:53.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:26:54.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:54.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:54.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:52 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:52' 2026-02-18T06:26:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:54.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:54.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:54.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:54.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:54.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:54.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:54.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:54.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:52' 2026-02-18T06:26:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:26:52' 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:52' 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:54.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:54.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:26:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:26:52' 2026-02-18T06:26:54.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:54.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:54.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:52' 2026-02-18T06:26:54.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:54.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:54.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:54.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:54.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:54.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:54.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:54.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:54.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:26:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:55.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:55.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:55.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:55.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:55.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:52 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:26:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:26:55.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:52' 2026-02-18T06:26:55.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:26:55.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:26:55.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:26:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:26:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:26:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:26:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:55.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:26:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:26:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:26:55.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:26:55.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:52' 2026-02-18T06:26:55.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:26:52' 2026-02-18T06:26:55.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:55.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:55.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:55.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:26:55.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:52' 2026-02-18T06:26:55.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:26:55.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:26:55.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:26:55.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:26:52cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:26:52' 2026-02-18T06:26:55.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:26:55.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:26:55.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:52' 2026-02-18T06:26:55.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:26:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:26:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:26:55.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:26:55.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:26:55.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:55.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:55.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:26:55.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:55.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:55.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:55.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:26:55.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:26:55.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:26:55.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:26:55.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:26:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:56.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:56.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:56.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:56.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:56.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:56.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:26:56.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:26:56.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:56.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:26:58.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:26:58.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:26:58.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:26:58.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:58.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:26:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:26:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:58.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:58.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:58.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:58.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:26:58.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:26:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:26:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:26:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:26:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:26:58.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:26:58.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:26:58.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:58.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:26:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:26:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:26:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:26:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:26:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:26:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:26:58.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:26:58.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:58.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:58.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:58.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:26:58.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:26:58.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:26:58.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:26:58.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:58.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:26:58.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:26:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:26:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:26:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:26:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:26:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:26:58.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:58.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:58.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:26:58.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:26:59.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:59.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:59.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:59.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:26:59.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:59.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=43, entry_tid=0' 2026-02-18T06:26:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:26:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:26:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:26:59.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:59.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:26:59.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:26:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:26:59.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:26:59.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:59.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:26:59.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:26:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=43, entry_tid=0' 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=43, entry_tid=0' -a 'object_number=0, tag_tid=43, entry_tid=0' = 'object_number=0, tag_tid=43, entry_tid=0' 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:26:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:27:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:00.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:00.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:57 2026-02-18T06:27:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2026-02-18T06:27:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:57' 2026-02-18T06:27:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:27:00.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:27:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:00.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:27:00.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:00.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:00.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:00.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:00.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:57' 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:26:57' 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:57' 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:00.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:26:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}2026-02-18 06:26:57' 2026-02-18T06:27:00.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:00.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:00.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:57' 2026-02-18T06:27:00.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:00.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:27:00.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:00.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:27:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:00.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:00.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:01.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:27:01.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:01.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:01.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:01.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2026-02-18T06:27:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:57 2026-02-18T06:27:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:26:57' 2026-02-18T06:27:01.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:27:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:27:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2026-02-18T06:27:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:01.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:01.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:01.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:01.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:01.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:57' 2026-02-18T06:27:01.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:26:57' 2026-02-18T06:27:01.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:26:57' 2026-02-18T06:27:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}43444348mirror.0trial1302026-02-18 06:26:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:26:57' 2026-02-18T06:27:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:01.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:26:57' 2026-02-18T06:27:01.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:27:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:27:01.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:27:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:27:01.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:01.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:01.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:01.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:27:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:27:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:27:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:27:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:02.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:02.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:02.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:02.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:02.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:02.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:02.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:02.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:02.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:02.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:27:02.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:02.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:03.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:27:03.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:02 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:02' 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:03.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:03.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:03.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:02' 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:27:02' 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:02' 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:03.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:03.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:27:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:27:02' 2026-02-18T06:27:03.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:03.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:02' 2026-02-18T06:27:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:03.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:03.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:03.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:03.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:03.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:03.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:04.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:27:04.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:04.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:02 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:02' 2026-02-18T06:27:04.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:04.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:04.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:04.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:04.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:04.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:04.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:04.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:04.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:02' 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:27:02' 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:02' 2026-02-18T06:27:04.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:04.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:04.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:04.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:27:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:27:02' 2026-02-18T06:27:04.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:04.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:04.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:02' 2026-02-18T06:27:04.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:04.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:04.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:04.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:04.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:27:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:27:04.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:27:04.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:27:04.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:04.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:04.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:05.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:27:05.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:05.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:05.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:05.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:05.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:05.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:05.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:05.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:05.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:05.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:05.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:05.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:05.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:05.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:05.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:05.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:05.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:27:05.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:05.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:07.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:27:07.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:07.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:07.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:07.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:07.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:07.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:07.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:07.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:07.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:07.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:07.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:07.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:07.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:07.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:07.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:07.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:27:07.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:07.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:07.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:07.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:27:07.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:27:07.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:27:07.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:27:07.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:27:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:27:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:27:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:27:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:27:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:07.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:07.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:07.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:07.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:07.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:07.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:27:07.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:07.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:27:07.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:07.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:07.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:07.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:27:07.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:07.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:07.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:07.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:07.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:07.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:07.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:07.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=44, entry_tid=0' 2026-02-18T06:27:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:27:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:07.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:27:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:27:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:07.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:07.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:07.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:27:07.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:08.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:08.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=44, entry_tid=0' 2026-02-18T06:27:08.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=44, entry_tid=0' -a 'object_number=0, tag_tid=44, entry_tid=0' = 'object_number=0, tag_tid=44, entry_tid=0' 2026-02-18T06:27:08.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:08.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:08.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:09.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:09.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:07 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2026-02-18T06:27:09.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:07' 2026-02-18T06:27:09.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:27:09.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:27:09.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:09.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:27:09.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:09.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:09.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:09.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:09.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:09.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:09.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:09.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:07' 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:27:07' 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:07' 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:09.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:09.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:27:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}2026-02-18 06:27:07' 2026-02-18T06:27:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:09.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:09.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:07' 2026-02-18T06:27:09.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:27:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:09.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:27:09.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:27:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:09.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:27:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:07 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:07' 2026-02-18T06:27:10.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:27:10.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:27:10.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2026-02-18T06:27:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:10.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:10.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:10.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:10.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:10.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:07' 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:27:07' 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:07' 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:10.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:10.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}45714576mirror.0trial1302026-02-18 06:27:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:27:07' 2026-02-18T06:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:10.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:07' 2026-02-18T06:27:10.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:10.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:27:10.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:10.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:27:10.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:27:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:27:10.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:27:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:27:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:27:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:11.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:11.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:11.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:11.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:11.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:11.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:11.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:11.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:11.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:11.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:11.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:11.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:11.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:11.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:11.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:11.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:11.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:11.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:11.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:11.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:11.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:27:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:11.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:13.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:13.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:13.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:13.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:13.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:13.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:13.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:13.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:13.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:13.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:13.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:14.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:14.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:12 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:12' 2026-02-18T06:27:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:14.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:12' 2026-02-18T06:27:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:27:12' 2026-02-18T06:27:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:12' 2026-02-18T06:27:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:14.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:14.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:27:12cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:27:12' 2026-02-18T06:27:14.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:14.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:14.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:12' 2026-02-18T06:27:14.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:14.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:14.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:14.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:14.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:14.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:14.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:15.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:27:15.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:15.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:12 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:15.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:15.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:15.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:12' 2026-02-18T06:27:15.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:15.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:15.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:15.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:15.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:15.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:15.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:15.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:15.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:12' 2026-02-18T06:27:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:27:12' 2026-02-18T06:27:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:12' 2026-02-18T06:27:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:15.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:27:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:27:12' 2026-02-18T06:27:15.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:15.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:15.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:12' 2026-02-18T06:27:15.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:15.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:15.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:15.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:15.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:15.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:15.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:15.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:27:15.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:15.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:15.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:27:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:27:15.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:27:15.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:15.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:27:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:16.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:16.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:16.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:16.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:16.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:16.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:16.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:16.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:16.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:16.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:16.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:16.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:27:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:18.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:18.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:18.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:18.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:18.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:18.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:18.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:18.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:18.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:18.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:18.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:18.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:18.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:18.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:18.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:27:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:18.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:27:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:27:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:27:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:27:18.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:27:18.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:27:18.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:18.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:27:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:27:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:27:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:27:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:27:18.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:18.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:18.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:18.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:18.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:18.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:18.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:27:18.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:18.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:18.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:18.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:27:18.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:18.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:18.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:18.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:18.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=48, entry_tid=0' 2026-02-18T06:27:18.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:27:18.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:18.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:18.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:18.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:27:18.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:18.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:18.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:18.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:27:18.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:18.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:18.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:18.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:18.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:27:18.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:18.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:18.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:18.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:18.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:18.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=48, entry_tid=0' 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=48, entry_tid=0' -a 'object_number=0, tag_tid=48, entry_tid=0' = 'object_number=0, tag_tid=48, entry_tid=0' 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:18.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:27:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:19.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:20.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:20.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:20.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:20.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:20.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:20.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:17 2026-02-18T06:27:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2026-02-18T06:27:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:17' 2026-02-18T06:27:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:27:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:27:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:27:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:20.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:20.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:20.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:17' 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:27:17' 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:17' 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:20.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:27:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}2026-02-18 06:27:17' 2026-02-18T06:27:20.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:20.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:20.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:17' 2026-02-18T06:27:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:27:20.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:20.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:27:20.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:20.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:20.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:21.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:21.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:17 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:17' 2026-02-18T06:27:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:27:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:27:21.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2026-02-18T06:27:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:21.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:21.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:21.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:21.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:17' 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:27:17' 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:17' 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:21.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}43444348mirror.0trial1302026-02-18 06:27:17cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:27:17' 2026-02-18T06:27:21.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:21.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:21.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:17' 2026-02-18T06:27:21.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:21.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:27:21.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:21.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:27:21.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:27:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:27:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:27:21.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:27:21.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:27:21.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:21.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:22.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:22.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:22.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:22.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:22.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:22.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:22.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:22.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:22.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:22.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:22.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:22.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:22.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:22.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:22.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:22.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:22.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:22.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:27:22.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:22.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:24.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:24.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:24.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:24.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:24.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:24.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:24.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:24.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:24.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:24.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:24.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:24.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:24.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:24.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:24.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:24.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:24.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:27:24.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:24.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:27:24.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:24.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:25.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:22 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:25.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:22' 2026-02-18T06:27:25.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:25.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:25.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:25.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:25.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:25.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:25.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:25.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:25.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:25.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:25.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:25.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:22' 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:27:22' 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:22' 2026-02-18T06:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:25.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:25.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:27:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:27:22' 2026-02-18T06:27:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:22' 2026-02-18T06:27:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:25.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:25.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:25.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:25.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:25.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:25.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:25.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:25.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:25.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:26.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:27:26.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:26.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:26.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:26.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:26.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:26.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:26.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:26.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:26.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:22 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:26.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:26.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:26.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:26.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:22' 2026-02-18T06:27:26.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:26.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:26.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:26.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:26.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:26.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:26.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:26.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:26.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:26.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:26.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:22' 2026-02-18T06:27:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:27:22' 2026-02-18T06:27:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:26.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:26.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:26.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:22' 2026-02-18T06:27:26.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:26.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:26.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:26.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:27:22cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:27:22' 2026-02-18T06:27:26.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:26.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:26.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:22' 2026-02-18T06:27:26.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:26.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:26.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:26.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:26.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:26.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:27:26.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:27:26.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:27:26.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:26.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:26.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:26.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:27.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:27:27.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:27.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:27.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:27.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:27.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:27.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:27.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:27.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:27.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:27.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:27.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:27.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:27.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:27.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:27.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:27.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:27.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:27.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:27.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:27.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:27:27.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:27:27.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:27:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:27:27.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:27.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:27:27.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:27.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:27:27.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:27.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:27:27.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:27:27.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:27:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:27:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:27:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:27.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:27.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:27.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:27.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:27:27.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:27:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:27:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:27.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:27.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:27.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:27.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:27.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:27.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:27.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=49, entry_tid=0' 2026-02-18T06:27:27.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:27:27.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:27.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:27.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:27.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:27:27.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:27.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:27.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:27.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:27:27.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:27.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:27.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:27.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:27.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:27:27.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:27.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:27.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:27.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:27.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:27.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:27.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=49, entry_tid=0' 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=49, entry_tid=0' -a 'object_number=0, tag_tid=49, entry_tid=0' = 'object_number=0, tag_tid=49, entry_tid=0' 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:28.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:28.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:27 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:27' 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:27:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:28.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:28.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:28.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:28.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:28.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:28.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:27' 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:27:27' 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:27' 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:28.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:27:27cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}2026-02-18 06:27:27' 2026-02-18T06:27:28.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:28.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:28.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:27' 2026-02-18T06:27:28.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:27:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:27:28.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:28.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:28.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:29.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:27:29.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:30.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:27 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:27' 2026-02-18T06:27:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:27:30.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:27:30.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2026-02-18T06:27:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:30.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:30.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:27' 2026-02-18T06:27:30.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:27:27' 2026-02-18T06:27:30.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:30.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:30.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:30.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:30.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:27' 2026-02-18T06:27:30.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:30.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:30.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}45714576mirror.0trial1302026-02-18 06:27:27cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:27:27' 2026-02-18T06:27:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:27' 2026-02-18T06:27:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:30.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:27:30.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:30.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:27:30.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:30.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:27:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:30.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:27:30.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:30.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:30.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:27:30.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:30.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:30.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:31.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:31.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:31.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:31.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:31.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:31.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:31.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:31.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:31.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:31.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:31.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:31.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:31.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:31.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:31.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:27:31.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:31.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:33.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:33.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:33.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:33.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:33.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:33.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:33.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:33.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:33.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:33.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:34.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:27:34.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:34.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:34.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:34.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:32 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:34.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:34.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:32' 2026-02-18T06:27:34.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:34.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:34.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:34.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:34.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:34.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:34.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:34.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:34.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:32' 2026-02-18T06:27:34.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:27:32' 2026-02-18T06:27:34.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:34.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:34.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:34.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:34.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:32' 2026-02-18T06:27:34.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:34.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:34.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:27:32cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:27:32' 2026-02-18T06:27:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:34.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:32' 2026-02-18T06:27:34.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:34.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:34.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:34.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:35.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:32 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:32' 2026-02-18T06:27:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:35.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:35.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:32' 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:27:32' 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:32' 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:27:32cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:27:32' 2026-02-18T06:27:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:32' 2026-02-18T06:27:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:35.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:35.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:35.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:35.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:27:35.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:35.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:27:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:27:35.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:27:35.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:35.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:35.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:27:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:36.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:36.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:36.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:36.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:36.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:36.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:36.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:36.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:36.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:36.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:36.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:36.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:27:36.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:36.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:38.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:38.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:38.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:38.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:38.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:38.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:38.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:38.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:38.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:38.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:38.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:38.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:38.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:38.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:27:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:27:38.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:27:38.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:27:38.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:27:38.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:27:38.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:38.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:27:38.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:38.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:38.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:38.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:38.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:38.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:38.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:27:38.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:38.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:27:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:38.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:38.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:38.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=53, entry_tid=0' 2026-02-18T06:27:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:27:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:38.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:38.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:27:38.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:38.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:38.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:38.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:38.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:38.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:27:38.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=53, entry_tid=0' 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=53, entry_tid=0' -a 'object_number=0, tag_tid=53, entry_tid=0' = 'object_number=0, tag_tid=53, entry_tid=0' 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:38.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:38.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:38.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:38.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:39.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:27:39.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:39.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:39.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:39.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:39.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:39.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:39.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:39.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:39.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:39.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:37 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:37' 2026-02-18T06:27:39.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:27:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:27:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:39.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:27:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:37' 2026-02-18T06:27:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:27:37' 2026-02-18T06:27:39.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:37' 2026-02-18T06:27:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:27:37cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}2026-02-18 06:27:37' 2026-02-18T06:27:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:39.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:39.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:37' 2026-02-18T06:27:39.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:39.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:27:39.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:39.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:27:39.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:27:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:27:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:37 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:40.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:40.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:40.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:37' 2026-02-18T06:27:40.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:27:40.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:27:40.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:40.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2026-02-18T06:27:40.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:40.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:40.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:40.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:40.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:40.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:37' 2026-02-18T06:27:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:27:37' 2026-02-18T06:27:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:40.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:40.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:40.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:37' 2026-02-18T06:27:40.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:40.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:40.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:41.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}43444348mirror.0trial1302026-02-18 06:27:37cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:27:37' 2026-02-18T06:27:41.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:41.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:41.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:37' 2026-02-18T06:27:41.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:41.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:27:41.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:41.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:27:41.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:41.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:41.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:41.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:27:41.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:27:41.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:41.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:41.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:41.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:27:41.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:41.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:42.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:42.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:42.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:42.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:42.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:42.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:42.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:42.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:42.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:42.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:42.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:42.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:42.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:42.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:42.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:42.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:42.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:42.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:42.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:42.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:42.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:27:42.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:42.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:44.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:27:44.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:44.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:44.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:44.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:44.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:44.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:44.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:27:44.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:27:44.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:44.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:44.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:27:44.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:44.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:44.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:44.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:44.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:27:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:27:44.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:27:44.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:27:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:27:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:44.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:44.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:44.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:44.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:45.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:45.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:45.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:45.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:45.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:42 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:42' 2026-02-18T06:27:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:45.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:45.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:45.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:45.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:45.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:45.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:45.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:45.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:45.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:45.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:45.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:42' 2026-02-18T06:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:27:42' 2026-02-18T06:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:42' 2026-02-18T06:27:45.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:45.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:45.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:27:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:27:42' 2026-02-18T06:27:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:45.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:45.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:42' 2026-02-18T06:27:45.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:45.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:45.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:45.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:45.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:45.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:45.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:45.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:27:45.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:45.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:45.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:45.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:27:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:46.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:42 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:42' 2026-02-18T06:27:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:46.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:46.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:46.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:46.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:46.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:46.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:42' 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:27:42' 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:42' 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:46.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:27:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:27:42' 2026-02-18T06:27:46.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:46.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:46.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:42' 2026-02-18T06:27:46.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:46.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:27:46.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:27:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:27:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:27:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:27:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:46.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:27:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:46.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:27:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:47.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:47.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:47.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:47.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:47.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:47.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:47.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:27:47.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:47.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:27:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:27:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:49.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:49.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:49.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:49.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:49.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:27:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:27:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:27:49.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:27:49.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:27:49.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:27:49.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:27:49.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:49.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:27:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:27:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:27:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:27:49.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:49.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:27:49.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:49.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:49.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:27:49.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:27:49.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:49.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:49.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:49.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:49.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:49.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:49.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:27:49.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:27:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:27:49.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:27:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:27:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:50.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:50.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:50.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=54, entry_tid=0' 2026-02-18T06:27:50.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:27:50.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:50.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:27:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:27:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:27:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:27:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:27:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=54, entry_tid=0' 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=54, entry_tid=0' -a 'object_number=0, tag_tid=54, entry_tid=0' = 'object_number=0, tag_tid=54, entry_tid=0' 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:50.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:47 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:51.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2026-02-18T06:27:51.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:47' 2026-02-18T06:27:51.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:27:51.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:27:51.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:51.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:27:51.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:51.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:51.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:51.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:51.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:51.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:47' 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:27:47' 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:47' 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:27:47cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}2026-02-18 06:27:47' 2026-02-18T06:27:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:51.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:47' 2026-02-18T06:27:51.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:27:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:27:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:52.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:27:52.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:52.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:52.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:52.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:47 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:47' 2026-02-18T06:27:52.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:27:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:27:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2026-02-18T06:27:52.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:52.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:52.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:52.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:47' 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:27:47' 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:47' 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:52.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}45714576mirror.0trial1302026-02-18 06:27:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:27:47' 2026-02-18T06:27:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:52.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:52.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:47' 2026-02-18T06:27:52.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:52.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:27:52.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:27:52.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:27:52.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:27:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:27:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:52.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:27:52.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:27:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:27:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:27:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:53.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:53.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:53.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:53.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:53.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:53.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:53.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:53.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:53.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:53.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:53.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:53.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:53.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:53.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:53.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:53.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:53.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:53.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:53.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:53.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:53.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:53.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:53.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:53.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:53.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:53.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:53.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:53.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:53.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:53.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:53.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:53.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:53.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:27:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:27:53.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:27:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:27:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:27:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:53.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:27:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:27:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:27:55.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:27:55.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:27:55.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:27:55.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:27:55.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:27:55.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:27:55.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:27:55.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:27:55.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:55.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:55.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:27:55.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:27:55.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:27:55.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:27:55.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:27:55.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:27:55.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:27:55.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:55.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:55.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:56.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:52 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:27:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:53' 2026-02-18T06:27:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:56.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:56.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:56.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:56.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:56.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:56.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:56.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:56.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:56.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:56.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:56.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:52' 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:27:52' 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:52' 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:56.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:27:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedstopped2026-02-18 06:27:53' 2026-02-18T06:27:56.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:56.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:56.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:52' 2026-02-18T06:27:56.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:56.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:56.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:56.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:56.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:56.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:56.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:56.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:27:56.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:27:57.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:27:57.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:53 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:57' 2026-02-18T06:27:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:27:57.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:59.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:57 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:27:57' 2026-02-18T06:27:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:27:59.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:27:59.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:27:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:27:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:27:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:27:59.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:59.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:27:59.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:27:59.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:27:59.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:57' 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:27:57' 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:27:57' 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:27:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:27:59.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:27:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:27:57' 2026-02-18T06:27:59.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:27:59.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:27:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:27:57' 2026-02-18T06:27:59.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:27:59.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:27:59.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:27:59.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:27:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:27:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:28:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:00.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:00.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:00.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:00.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:00.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:00.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:00.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:00.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:00.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:00.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:00.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:00.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:00.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:28:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:02.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:02.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:02.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:02.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:02.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:02.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:02.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:02.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:02.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:28:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:28:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:02.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:28:02.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:28:02.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:02.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:28:03.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:28:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:03.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:03.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:28:03.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:03.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:03.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:03.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=55, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=58, entry_tid=0' 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:28:03.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:03.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:03.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:03.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:28:03.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=55, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=55, entry_tid=0' 2026-02-18T06:28:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=58, entry_tid=0' -a 'object_number=0, tag_tid=58, entry_tid=0' = 'object_number=0, tag_tid=55, entry_tid=0' 2026-02-18T06:28:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=55, entry_tid=0' '!=' '' 2026-02-18T06:28:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:28:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=55, entry_tid=0' = '' 2026-02-18T06:28:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=0, tag_tid=55, entry_tid=0' 2026-02-18T06:28:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=58, entry_tid=0' 2026-02-18T06:28:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:28:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:28:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=58 2026-02-18T06:28:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=55, entry_tid=0' 2026-02-18T06:28:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:28:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:28:03.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=55 2026-02-18T06:28:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=58, entry_tid=0' 2026-02-18T06:28:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:28:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:28:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=0 2026-02-18T06:28:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=55, entry_tid=0' 2026-02-18T06:28:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:28:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:28:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=0 2026-02-18T06:28:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 58 = 55 -a 0 -le 0 2026-02-18T06:28:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:28:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:28:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:28:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:28:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:03.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:03.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:03.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:03.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:28:03.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:03.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:28:03.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:03.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:03.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:03.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:28:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=58, entry_tid=0' 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:03.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:03.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:28:03.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:03.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:03.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:03.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:28:03.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:03.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=58, entry_tid=0' 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=58, entry_tid=0' -a 'object_number=0, tag_tid=58, entry_tid=0' = 'object_number=0, tag_tid=58, entry_tid=0' 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:03.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:04.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:04.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:04.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:02 2026-02-18T06:28:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:04.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2026-02-18T06:28:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:02' 2026-02-18T06:28:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:28:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:28:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:28:04.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:04.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:04.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:04.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:04.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:04.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:02' 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:28:02' 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:04.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:02' 2026-02-18T06:28:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:04.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:04.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:28:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}2026-02-18 06:28:02' 2026-02-18T06:28:04.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:04.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:04.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:02' 2026-02-18T06:28:04.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:04.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:28:04.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:04.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:28:04.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:28:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:04.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:04.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:04.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:04.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:05.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:28:05.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:05.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:05.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:05.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:05.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:05.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:05.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:05.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:05.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:05.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:05.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:05.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:05.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:02 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:02' 2026-02-18T06:28:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:28:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:28:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:05.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2026-02-18T06:28:05.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:05.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:05.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:05.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:05.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:05.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:05.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:05.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:02' 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:28:02' 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:02' 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:05.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}43444348mirror.0trial1302026-02-18 06:28:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:28:02' 2026-02-18T06:28:05.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:05.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:05.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:02' 2026-02-18T06:28:05.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:05.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:28:05.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:05.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:05.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:28:05.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:28:05.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:28:05.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:05.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:05.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:05.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:28:05.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:05.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:05.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:05.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:28:05.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:05.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:05.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:06.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:28:06.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:06.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:06.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:06.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:06.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:06.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:06.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:06.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:06.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:06.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:06.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:06.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:06.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:28:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:06.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:08.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:08.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:08.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:08.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:08.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:08.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:08.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:08.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:08.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:08.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:09.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:28:09.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:09.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:09.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:09.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:07 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:07' 2026-02-18T06:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:09.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:09.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:09.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:09.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:09.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:07' 2026-02-18T06:28:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:28:07' 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:07' 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:10.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:28:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:28:07' 2026-02-18T06:28:10.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:10.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:10.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:07' 2026-02-18T06:28:10.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:10.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:10.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:10.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:11.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:28:11.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:11.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:11.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:11.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:11.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:11.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:11.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:11.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:11.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:07 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:11.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:11.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:11.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:07' 2026-02-18T06:28:11.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:11.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:11.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:11.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:11.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:11.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:11.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:11.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:07' 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:28:07' 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:07' 2026-02-18T06:28:11.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:11.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:11.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:11.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:28:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:28:07' 2026-02-18T06:28:11.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:11.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:11.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:07' 2026-02-18T06:28:11.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:11.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:11.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:11.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:11.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:11.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:28:11.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:11.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:11.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:11.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:28:11.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:28:11.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:28:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:28:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:28:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:28:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:11.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:28:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:12.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:12.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:12.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:12.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:12.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:12.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:12.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:28:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:14.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:14.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:14.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:14.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:14.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:14.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:14.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:14.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:28:14.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:28:14.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:28:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:28:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:28:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:28:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:28:14.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:14.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:28:14.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:14.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:28:14.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:28:14.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:14.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:14.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:14.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:14.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:28:14.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:14.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:28:14.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:14.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:14.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:14.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:28:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:14.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:14.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:14.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:14.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=59, entry_tid=0' 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:14.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:28:14.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:14.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:14.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:14.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:14.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:14.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=59, entry_tid=0' 2026-02-18T06:28:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=59, entry_tid=0' -a 'object_number=0, tag_tid=59, entry_tid=0' = 'object_number=0, tag_tid=59, entry_tid=0' 2026-02-18T06:28:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:28:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:15.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:15.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:15.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:15.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:12 2026-02-18T06:28:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2026-02-18T06:28:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:12' 2026-02-18T06:28:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:28:15.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:28:15.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:15.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:28:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:15.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:15.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:15.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:12' 2026-02-18T06:28:15.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:28:12' 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:12' 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:15.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:28:12cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}2026-02-18 06:28:12' 2026-02-18T06:28:15.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:15.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:12' 2026-02-18T06:28:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:15.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:28:15.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:15.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:28:15.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:28:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:16.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:16.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:16.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:16.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:16.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:16.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:16.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:12 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:16.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:16.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:16.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:12' 2026-02-18T06:28:16.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:28:16.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:28:16.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:16.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2026-02-18T06:28:16.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:16.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:16.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:16.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:16.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:16.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:12' 2026-02-18T06:28:16.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:28:12' 2026-02-18T06:28:16.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:16.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:16.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:16.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:16.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:12' 2026-02-18T06:28:16.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:16.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:16.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:16.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}45714576mirror.0trial1302026-02-18 06:28:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:28:12' 2026-02-18T06:28:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:12' 2026-02-18T06:28:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:16.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:28:16.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:16.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:28:16.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:28:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:28:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:28:16.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:16.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:17.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:28:17.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:17.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:17.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:17.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:17.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:17.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:17.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:17.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:17.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:17.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:17.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:17.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:17.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:17.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:17.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:17.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:17.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:17.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:17.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:28:17.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:17.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:19.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:28:19.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:19.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:19.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:19.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:19.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:19.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:19.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:19.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:19.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:19.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:19.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:19.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:19.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:19.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:28:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:17 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:28:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:18' 2026-02-18T06:28:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:20.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:20.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:17' 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:28:17' 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:17' 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:20.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:21.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:28:17cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedstopped2026-02-18 06:28:18' 2026-02-18T06:28:21.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:21.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:21.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:17' 2026-02-18T06:28:21.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:21.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:21.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:28:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:21.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:21.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:21.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:21.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:22.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:28:22.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:22.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:22.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:22.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:22.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:22.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:18 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:17' 2026-02-18T06:28:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:22.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:28:22.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:22.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:28:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:24.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:22 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:22' 2026-02-18T06:28:24.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:24.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:24.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:24.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:24.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:24.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:24.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:24.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:22' 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:28:22' 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:22' 2026-02-18T06:28:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:24.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:24.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:28:22cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:28:22' 2026-02-18T06:28:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:22' 2026-02-18T06:28:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:24.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:24.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:28:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:28:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:28:24.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:24.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:28:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:25.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:25.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:25.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:25.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:25.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:25.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:25.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:25.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:25.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:25.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:25.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:27.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:28:27.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:27.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:27.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:27.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:27.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:27.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:27.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:27.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:27.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:27.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:27.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:27.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:27.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:27.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:27.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:27.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:28:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:28:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:31.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:31.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:31.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:31.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:31.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:31.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:31.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:31.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:31.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:31.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:31.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:28:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:28:31.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:28:31.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:28:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:28:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:28:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:28:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:28:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:31.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:28:31.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:28:31.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:31.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:28:31.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:31.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:31.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:31.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:31.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:31.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:31.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=63, entry_tid=0' 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:28:31.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=63, entry_tid=0' 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=63, entry_tid=0' -a 'object_number=0, tag_tid=63, entry_tid=0' = 'object_number=0, tag_tid=63, entry_tid=0' 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:31.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:31.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:32.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:28:32.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:32.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:32.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:32.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:32.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:32.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:32.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:32.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:32 2026-02-18T06:28:32.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:32.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:32.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:32.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2026-02-18T06:28:32.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:27' 2026-02-18T06:28:32.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:28:32.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:28:32.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:32.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:28:32.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:32.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:32.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:32.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:32.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:32.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:32.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:32.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:32.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:32' 2026-02-18T06:28:32.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:28:32' 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:32' 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:32.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:32.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:28:32cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}2026-02-18 06:28:27' 2026-02-18T06:28:32.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:32.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:32.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:32' 2026-02-18T06:28:32.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:32.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:28:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:28:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:32.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:33.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:28:33.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:33.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:33.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2026-02-18T06:28:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:27 2026-02-18T06:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:32' 2026-02-18T06:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:28:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:28:33.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:33.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2026-02-18T06:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:33.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:33.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:27' 2026-02-18T06:28:33.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:28:27' 2026-02-18T06:28:33.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:33.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:27' 2026-02-18T06:28:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}43444348mirror.0trial1302026-02-18 06:28:27cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:28:32' 2026-02-18T06:28:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:33.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:33.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:27' 2026-02-18T06:28:33.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:33.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:28:33.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:33.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:28:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:28:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:33.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:34.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:34.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:34.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:34.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:34.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:34.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:34.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:34.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:34.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:34.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:34.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:34.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:34.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:34.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:34.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:34.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:34.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:34.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:34.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:34.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:28:34.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:34.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:36.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:28:36.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:36.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:36.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:36.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:36.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:36.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:36.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:36.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:36.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:36.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:36.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:36.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:36.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:36.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:36.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:36.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:36.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:36.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:37.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:37.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:37.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:37.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:37.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:38.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:38.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:37 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:37' 2026-02-18T06:28:38.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:37' 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:28:37' 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:37' 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:28:37cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:28:37' 2026-02-18T06:28:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:38.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:37' 2026-02-18T06:28:38.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:38.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:38.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:38.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:38.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:28:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:38.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:38.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:38.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:38.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:28:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:39.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:37 2026-02-18T06:28:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:37' 2026-02-18T06:28:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:39.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:39.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:39.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:39.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:39.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:37' 2026-02-18T06:28:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:28:37' 2026-02-18T06:28:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:37' 2026-02-18T06:28:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:39.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:39.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:28:37cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:28:37' 2026-02-18T06:28:39.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:39.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:37' 2026-02-18T06:28:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:28:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:28:39.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:40.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:40.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:40.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:40.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:40.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:40.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:40.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:40.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:40.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:40.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:40.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:40.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:40.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:40.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:40.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:40.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:40.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:28:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:42.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:42.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:42.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:42.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:42.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:42.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:42.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:42.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:42.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:42.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:42.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:42.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:42.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:42.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:42.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:42.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:28:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:28:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:46.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:46.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:46.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:46.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:46.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:46.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:46.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:46.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:46.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:46.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:46.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:28:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:28:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:28:46.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:28:46.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:46.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:46.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:28:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:28:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:46.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:28:46.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:28:46.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:46.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:46.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:46.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:46.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:46.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:46.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:28:46.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:28:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:28:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:28:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:28:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:46.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:46.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:46.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:46.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=64, entry_tid=0' 2026-02-18T06:28:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:28:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:28:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:28:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:28:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:28:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:28:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:28:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:28:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:28:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:28:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:28:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:28:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:28:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:28:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=64, entry_tid=0' 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=64, entry_tid=0' -a 'object_number=0, tag_tid=64, entry_tid=0' = 'object_number=0, tag_tid=64, entry_tid=0' 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:46.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:28:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:47.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:47.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:47.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:47.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:47.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:47.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:47.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:47.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:47 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:42' 2026-02-18T06:28:47.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:28:47.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:28:47.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:47.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:28:47.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:47.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:47.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:47.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:47.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:47.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:47.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:47.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:47.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:47' 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:28:47' 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:47' 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:47.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:47.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:28:47cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}2026-02-18 06:28:42' 2026-02-18T06:28:47.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:47.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:47.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:47' 2026-02-18T06:28:47.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:47.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:28:47.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:28:47.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:28:47.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:28:47.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:47.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:48.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:48.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:48.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:42 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:47' 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:28:48.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:48.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2026-02-18T06:28:48.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:48.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:48.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:48.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:48.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:42' 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:28:42' 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:42' 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:48.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}45714576mirror.0trial1302026-02-18 06:28:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:28:47' 2026-02-18T06:28:48.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:48.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:48.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:42' 2026-02-18T06:28:48.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:48.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:28:48.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:28:48.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:28:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:28:48.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:28:48.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:28:48.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:48.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:48.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:48.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:28:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:49.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:28:49.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:28:49.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:49.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:28:49.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:28:49.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:28:49.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:49.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:28:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:28:51.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:51.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:28:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:28:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:28:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:28:51.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:28:51.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:28:51.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:51.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:51.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:28:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:28:52.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:52.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:52.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:52.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:52.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:52.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:28:52.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:53.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:28:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:52 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:52' 2026-02-18T06:28:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:53.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:53.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:53.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:52' 2026-02-18T06:28:53.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:28:52' 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:52' 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:53.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:28:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:28:52' 2026-02-18T06:28:53.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:53.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:53.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:52' 2026-02-18T06:28:53.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:53.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:53.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:53.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:54.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:28:54.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:54.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:54.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:54.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:54.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:28:54.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:28:54.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:54.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:54.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:28:54.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:28:54.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:28:54.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:54.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:52 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:52' 2026-02-18T06:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:28:54.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:28:54.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:28:54.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:28:54.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:28:54.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:54.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:28:54.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:54.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:28:54.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:28:54.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:28:54.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:28:54.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:52' 2026-02-18T06:28:54.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:28:52' 2026-02-18T06:28:54.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:54.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:28:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:52' 2026-02-18T06:28:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:28:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:28:54.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:28:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:28:52cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:28:52' 2026-02-18T06:28:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:28:54.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:28:54.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:52' 2026-02-18T06:28:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:28:54.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:28:54.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:28:54.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:28:54.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:28:54.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:54.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:28:54.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:28:54.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:28:54.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:54.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:54.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:28:54.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:28:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:28:54.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:28:54.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:54.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:54.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:54.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:28:54.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:54.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:54.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:54.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:28:54.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:28:54.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:54.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:28:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:28:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:55.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:55.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:55.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:55.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:55.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:28:57.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:28:57.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:28:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:28:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:28:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:28:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:28:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:28:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:28:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:28:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:28:57.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:28:57.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:28:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:28:57.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:57.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:57.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:28:57.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:28:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:28:57.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:28:57.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:28:57.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:28:57.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:28:57.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:28:57.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:29:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:29:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:01.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:01.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:01.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:01.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:01.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:01.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:01.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:01.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:01.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:01.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:29:01.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:01.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:29:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:29:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:29:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:01.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:29:01.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:29:01.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:01.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:01.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:01.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:01.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:01.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:01.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:01.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:01.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:01.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:01.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:01.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:01.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:01.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:01.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:01.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:01.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:01.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:01.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=68, entry_tid=0' 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:01.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:01.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:01.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:01.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:01.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:01.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:01.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=68, entry_tid=0' 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=68, entry_tid=0' -a 'object_number=0, tag_tid=68, entry_tid=0' = 'object_number=0, tag_tid=68, entry_tid=0' 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:01.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:01.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:01.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:01.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:01.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:02.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:02.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:02.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:02.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:02 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:57' 2026-02-18T06:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:29:02.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:29:02.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:02.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:02' 2026-02-18T06:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:29:02' 2026-02-18T06:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:02' 2026-02-18T06:29:02.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:02.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:02.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:29:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}2026-02-18 06:28:57' 2026-02-18T06:29:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:02' 2026-02-18T06:29:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:02.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:29:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:29:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:29:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:02.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:02.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:02.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:02.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:02.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:02.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:28:57 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:02' 2026-02-18T06:29:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:29:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:29:03.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:03.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2026-02-18T06:29:03.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:03.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:03.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:03.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:03.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:03.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:03.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:03.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:03.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:57' 2026-02-18T06:29:03.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:28:57' 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:28:57' 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:03.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}43444348mirror.0trial1302026-02-18 06:28:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:29:02' 2026-02-18T06:29:03.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:03.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:28:57' 2026-02-18T06:29:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:03.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:29:03.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:03.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:29:03.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:03.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:29:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:29:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:29:03.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:29:03.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:29:03.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:03.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:03.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:04.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:04.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:04.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:04.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:04.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:04.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:04.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:04.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:04.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:04.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:05.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:05.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:29:05.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:05.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:07.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:07.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:07.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:07.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:07.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:07.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:07.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:07.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:07.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:07.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:07.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:07.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:07.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:07.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:07.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:29:07.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:08.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:07 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:08.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:07' 2026-02-18T06:29:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:08.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:08.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:08.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:08.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:07' 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:29:07' 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:07' 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:08.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:08.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:29:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:29:07' 2026-02-18T06:29:08.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:08.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:08.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:07' 2026-02-18T06:29:08.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:08.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:08.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:08.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:08.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:29:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:08.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:08.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:08.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:08.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:08.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:08.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:08.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:09.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:29:09.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:09.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:09.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:09.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:09.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:09.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:07 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:07' 2026-02-18T06:29:09.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:09.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:09.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:09.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:09.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:09.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:09.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:09.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:09.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:07' 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:29:07' 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:07' 2026-02-18T06:29:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:29:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:29:07' 2026-02-18T06:29:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:09.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:07' 2026-02-18T06:29:09.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:09.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:09.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:09.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:09.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:29:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:29:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:29:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:10.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:29:10.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:10.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:10.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:10.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:10.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:10.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:10.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:10.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:10.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:10.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:10.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:10.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:29:10.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:10.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:12.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:12.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:12.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:12.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:12.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:12.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:12.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:12.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:12.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:12.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:29:12.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:12.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:29:16.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:29:16.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:16.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:16.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:16.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:16.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:16.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:16.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:16.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:16.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:16.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:16.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:16.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:16.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:16.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:16.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:16.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:16.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:16.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:16.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:16.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:16.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:16.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:16.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:16.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:29:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:29:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:29:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:29:16.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:16.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:29:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:29:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:16.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:16.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:29:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:29:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:16.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:16.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:29:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:16.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:16.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:29:16.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:16.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:16.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:16.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:16.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:16.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=69, entry_tid=0' 2026-02-18T06:29:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:29:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:16.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:16.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=69, entry_tid=0' 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=69, entry_tid=0' -a 'object_number=0, tag_tid=69, entry_tid=0' = 'object_number=0, tag_tid=69, entry_tid=0' 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:16.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:16.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:29:17.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:17.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:17 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:17.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2026-02-18T06:29:17.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:12' 2026-02-18T06:29:17.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:29:17.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:29:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:29:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:17' 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:29:17' 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:17' 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:17.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:29:17cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}2026-02-18 06:29:12' 2026-02-18T06:29:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:17.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:17.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:17' 2026-02-18T06:29:17.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:17.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:29:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:29:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:18.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:18.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:12 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:17' 2026-02-18T06:29:18.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:29:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:29:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:18.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2026-02-18T06:29:18.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:18.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:18.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:18.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:18.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:18.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:18.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:18.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:18.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:12' 2026-02-18T06:29:18.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:29:12' 2026-02-18T06:29:18.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:18.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:18.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:18.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:18.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:12' 2026-02-18T06:29:18.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:18.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:18.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:18.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}45714576mirror.0trial1302026-02-18 06:29:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:29:17' 2026-02-18T06:29:18.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:18.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:18.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:12' 2026-02-18T06:29:18.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:18.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:29:18.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:18.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:29:18.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:29:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:18.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:29:19.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:29:19.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:29:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:29:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:19.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:19.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:19.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:29:19.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:19.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:19.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:29:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:20.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:20.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:20.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:20.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:20.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:20.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:20.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:20.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:20.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:20.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:20.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:20.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:20.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:20.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:20.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:29:20.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:20.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:22.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:29:22.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:22.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:22.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:22.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:22.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:22.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:22.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:22.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:22.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:22.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:22.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:22.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:22.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:22.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:22.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:22.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:22.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:22.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:22.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:22.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:22.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:22.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:29:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:29:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:22.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:22.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:22.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:22.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:23.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:23.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:23.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:22 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:23.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:23.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:22' 2026-02-18T06:29:23.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:23.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:23.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:23.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:23.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:23.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:23.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:23.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:23.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:22' 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:29:22' 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:22' 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:23.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:23.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:29:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:29:22' 2026-02-18T06:29:23.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:23.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:23.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:22' 2026-02-18T06:29:23.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:23.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:23.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:23.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:23.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:29:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:23.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:24.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:29:24.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:24.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:24.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:24.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:24.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:24.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:22 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:22' 2026-02-18T06:29:24.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:24.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:24.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:24.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:24.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:24.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:24.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:22' 2026-02-18T06:29:24.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:29:22' 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:22' 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:24.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:29:22cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:29:22' 2026-02-18T06:29:24.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:24.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:22' 2026-02-18T06:29:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:29:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:29:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:29:24.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:25.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:29:25.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:25.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:25.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:25.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:25.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:25.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:25.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:25.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:25.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:25.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:25.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:25.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:25.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:25.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:25.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:25.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:29:25.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:25.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:27.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:29:27.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:27.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:27.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:27.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:27.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:27.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:27.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:27.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:27.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:27.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:27.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:27.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:27.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:27.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:27.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:27.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:29:27.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:27.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:29:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:27.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:27.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:29:27.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:29:27.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:27.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:27.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:27.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:29:27.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:27.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:27.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:27.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:27.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:27.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:27.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=70, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=73, entry_tid=0' 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:27.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:27.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:27.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:27.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:27.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:27.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=70, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=73, entry_tid=0' -a 'object_number=0, tag_tid=73, entry_tid=0' = 'object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=70, entry_tid=0' '!=' '' 2026-02-18T06:29:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:29:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=70, entry_tid=0' = '' 2026-02-18T06:29:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=73, entry_tid=0' 2026-02-18T06:29:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:29:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=73 2026-02-18T06:29:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:29:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=70 2026-02-18T06:29:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=73, entry_tid=0' 2026-02-18T06:29:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:29:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:28.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=0 2026-02-18T06:29:28.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:29:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=0 2026-02-18T06:29:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 73 = 70 -a 0 -le 0 2026-02-18T06:29:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:29:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:29:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:29:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:28.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:28.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:28.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:28.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:28.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:28.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:28.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:28.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:28.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:28.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:28.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=70, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=73, entry_tid=0' 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:28.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:28.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:28.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:28.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=70, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=73, entry_tid=0' -a 'object_number=0, tag_tid=73, entry_tid=0' = 'object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=70, entry_tid=0' '!=' 'object_number=0, tag_tid=70, entry_tid=0' 2026-02-18T06:29:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:29:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:29:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:28.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:28.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:28.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:28.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:28.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:28.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:28.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:28.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:28.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:28.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:28.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:28.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=73, entry_tid=0' 2026-02-18T06:29:28.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:29:28.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:28.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:28.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:28.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:28.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:28.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:28.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:28.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=73, entry_tid=0' 2026-02-18T06:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=73, entry_tid=0' -a 'object_number=0, tag_tid=73, entry_tid=0' = 'object_number=0, tag_tid=73, entry_tid=0' 2026-02-18T06:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:28.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:29.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:27 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:27' 2026-02-18T06:29:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:29:29.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:29:29.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:29.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:29:29.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:29.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:29.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:29.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:29.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:29.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:29.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:29.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:29.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:27' 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:29:27' 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:27' 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:29:27cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}2026-02-18 06:29:27' 2026-02-18T06:29:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:27' 2026-02-18T06:29:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:29.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:29:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:29:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:29.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:29.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:30.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:30.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:27 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:27' 2026-02-18T06:29:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:29:30.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:29:30.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:30.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2026-02-18T06:29:30.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:30.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:30.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:30.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:30.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:30.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:27' 2026-02-18T06:29:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:29:27' 2026-02-18T06:29:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:27' 2026-02-18T06:29:30.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:30.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:30.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}43444348mirror.0trial1302026-02-18 06:29:27cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:29:27' 2026-02-18T06:29:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:31.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:27' 2026-02-18T06:29:31.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:29:31.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:31.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:29:31.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:31.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:31.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:31.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:29:31.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:29:31.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:29:31.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:29:31.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:29:31.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:32.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:32.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:32.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:32.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:32.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:29:32.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:32.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:34.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:34.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:34.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:34.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:34.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:34.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:34.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:35.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:32 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:32' 2026-02-18T06:29:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:35.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:35.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:35.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:35.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:35.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:32' 2026-02-18T06:29:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:29:32' 2026-02-18T06:29:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:35.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:35.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:32' 2026-02-18T06:29:35.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:35.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:35.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:29:32cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:29:32' 2026-02-18T06:29:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:35.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:32' 2026-02-18T06:29:35.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:35.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:35.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:35.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:35.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:35.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:35.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:35.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:29:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:32 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:32' 2026-02-18T06:29:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:36.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:36.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:36.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:32' 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:29:32' 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:32' 2026-02-18T06:29:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:36.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:29:32cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:29:32' 2026-02-18T06:29:36.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:36.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:36.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:32' 2026-02-18T06:29:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:36.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:36.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:36.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:36.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:36.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:29:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:29:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:29:36.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:29:36.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:36.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:37.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:37.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:37.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:37.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:37.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:37.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:29:37.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:37.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:39.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:39.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:39.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:39.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:39.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:39.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:39.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:39.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:39.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:39.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:39.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:39.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:39.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:29:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:39.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:29:39.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:29:39.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:39.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:39.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:39.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:39.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:29:39.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:39.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:39.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:29:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:29:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:39.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:39.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:39.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:39.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:39.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:39.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:29:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:39.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:39.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:29:39.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:39.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:39.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:39.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:39.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:39.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:39.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:39.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=74, entry_tid=0' 2026-02-18T06:29:39.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:29:39.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:39.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:39.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:39.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:29:39.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:39.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:39.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:39.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:39.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:39.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:39.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:29:39.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:39.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:29:39.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=74, entry_tid=0' 2026-02-18T06:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=74, entry_tid=0' -a 'object_number=0, tag_tid=74, entry_tid=0' = 'object_number=0, tag_tid=74, entry_tid=0' 2026-02-18T06:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:39.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:40.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:29:40.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:40.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:40.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:40.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:40.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:37 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:37' 2026-02-18T06:29:40.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:29:40.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:29:40.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:40.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:40.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:40.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:40.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:40.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:40.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:37' 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:29:37' 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:37' 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:40.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:29:37cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}2026-02-18 06:29:37' 2026-02-18T06:29:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:37' 2026-02-18T06:29:40.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:29:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:29:40.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:29:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:41.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:41.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:41.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:41.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:37 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:37' 2026-02-18T06:29:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:29:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:29:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2026-02-18T06:29:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:41.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:41.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:41.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:41.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:41.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:41.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:41.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:37' 2026-02-18T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:29:37' 2026-02-18T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:37' 2026-02-18T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}45714576mirror.0trial1302026-02-18 06:29:37cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:29:37' 2026-02-18T06:29:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:37' 2026-02-18T06:29:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:42.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:29:42.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:42.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:29:42.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:29:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:42.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:42.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:29:42.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:29:42.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:29:42.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:42.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:29:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:43.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:43.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:43.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:43.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:29:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:29:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:29:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:29:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:45.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:29:45.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:45.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:45.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:29:45.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:45.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:45.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:45.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:45.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:29:45.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:46.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:46.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:42 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:43' 2026-02-18T06:29:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:46.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:46.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:46.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:46.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:46.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:46.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:46.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:46.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:42' 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:29:42' 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:42' 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:29:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedstopped2026-02-18 06:29:43' 2026-02-18T06:29:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:46.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:46.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:42' 2026-02-18T06:29:46.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:46.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:46.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:46.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:46.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:46.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:46.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:46.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:47.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:43 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:42' 2026-02-18T06:29:47.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:29:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:47.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:49.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:29:49.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:49.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:49.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:49.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:49.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:47 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:47' 2026-02-18T06:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:29:49.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:49.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:29:49.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:49.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:49.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:49.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:49.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:49.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:49.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:49.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:47' 2026-02-18T06:29:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:29:47' 2026-02-18T06:29:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:49.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:49.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:47' 2026-02-18T06:29:49.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:49.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:49.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:29:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:29:47' 2026-02-18T06:29:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:49.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:49.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:47' 2026-02-18T06:29:49.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:49.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:29:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:29:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:29:49.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:29:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:29:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:29:49.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:29:49.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:29:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:29:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:29:49.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:29:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:49.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:50.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:29:50.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:50.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:50.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:50.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:50.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:50.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:50.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:50.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:50.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:50.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:50.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:50.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:50.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:50.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:50.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:29:50.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:50.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:52.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:29:52.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:52.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:52.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:52.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:29:52.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:52.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:52.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:52.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:52.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:52.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:52.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:52.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:52.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:52.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:52.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:52.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:29:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:29:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:29:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:29:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:29:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:52.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:29:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:29:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:29:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:52.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:52.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:52.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:52.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:52.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:52.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:29:52.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:29:52.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:52.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:29:52.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:52.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:52.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:52.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:52.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:52.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:52.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:52.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:52.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:52.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:52.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:52.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:52.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=75, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:52.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:52.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=78, entry_tid=0' 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:52.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:52.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:52.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:52.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=75, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:52.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:52.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:52.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=78, entry_tid=0' -a 'object_number=0, tag_tid=78, entry_tid=0' = 'object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:52.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=75, entry_tid=0' '!=' '' 2026-02-18T06:29:52.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:29:52.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=75, entry_tid=0' = '' 2026-02-18T06:29:52.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:52.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=78, entry_tid=0' 2026-02-18T06:29:52.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:29:52.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:52.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=78 2026-02-18T06:29:52.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:52.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:29:52.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:52.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=75 2026-02-18T06:29:52.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=78, entry_tid=0' 2026-02-18T06:29:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:29:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=0 2026-02-18T06:29:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:29:52.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=0 2026-02-18T06:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 78 = 75 -a 0 -le 0 2026-02-18T06:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:53.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:53.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:53.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:53.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:53.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:53.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:53.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=75, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=78, entry_tid=0' 2026-02-18T06:29:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:29:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:53.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:53.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:53.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=75, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=78, entry_tid=0' -a 'object_number=0, tag_tid=78, entry_tid=0' = 'object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=0, tag_tid=75, entry_tid=0' '!=' 'object_number=0, tag_tid=75, entry_tid=0' 2026-02-18T06:29:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:29:53.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:29:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:29:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:29:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:29:53.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:53.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:53.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:29:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:29:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:29:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:29:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:53.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:53.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:53.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=78, entry_tid=0' 2026-02-18T06:29:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:29:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:29:53.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:29:53.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:29:53.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:29:53.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:29:53.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:29:53.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:29:53.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:29:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=78, entry_tid=0' 2026-02-18T06:29:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=78, entry_tid=0' -a 'object_number=0, tag_tid=78, entry_tid=0' = 'object_number=0, tag_tid=78, entry_tid=0' 2026-02-18T06:29:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:29:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:54.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:54.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:52 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:54.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2026-02-18T06:29:54.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:52' 2026-02-18T06:29:54.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:29:54.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:29:54.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:54.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:29:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:52' 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:29:52' 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:52' 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:54.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:29:52cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}2026-02-18 06:29:52' 2026-02-18T06:29:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:52' 2026-02-18T06:29:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:29:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:29:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:29:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:29:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:29:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:29:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:55.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:29:55.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:29:55.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:29:55.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:29:55.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:52 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:52' 2026-02-18T06:29:55.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:29:55.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:29:55.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:29:55.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2026-02-18T06:29:55.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:29:55.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:55.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:29:55.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:55.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:29:55.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:29:55.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:29:55.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:29:55.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:52' 2026-02-18T06:29:55.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:29:52' 2026-02-18T06:29:55.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:29:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:52' 2026-02-18T06:29:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:29:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:29:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:29:55.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}43444348mirror.0trial1302026-02-18 06:29:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:29:52' 2026-02-18T06:29:55.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:29:55.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:29:55.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:52' 2026-02-18T06:29:55.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:29:55.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:29:55.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:29:55.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:29:55.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:55.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:29:55.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:29:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:29:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:55.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:29:56.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:56.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:56.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:56.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:56.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:56.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:56.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:56.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:56.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:56.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:29:56.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:29:56.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:29:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:29:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:29:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:29:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:29:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:29:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:29:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:29:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:29:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:29:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:29:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:29:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:29:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:29:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:29:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:29:58.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:29:58.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:29:59.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:29:59.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:59.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:59.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:29:59.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:29:59.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:29:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:29:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:29:59.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:29:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:29:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:29:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:29:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:29:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:29:59.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:29:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:29:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:29:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:29:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:29:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:30:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:00.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:00.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:00.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:00.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:00.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:00.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:00.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:00.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:57 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:57' 2026-02-18T06:30:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:00.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:00.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:00.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:00.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:57' 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:29:57' 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:57' 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:00.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:00.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:29:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:29:57' 2026-02-18T06:30:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:57' 2026-02-18T06:30:00.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:00.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:00.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:00.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:00.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:30:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:01.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:57 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:29:57' 2026-02-18T06:30:01.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:01.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:01.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:01.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:57' 2026-02-18T06:30:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:29:57' 2026-02-18T06:30:01.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:29:57' 2026-02-18T06:30:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:01.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:29:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:29:57' 2026-02-18T06:30:01.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:01.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:01.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:29:57' 2026-02-18T06:30:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:30:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:30:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:30:01.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:30:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:30:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:01.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:01.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:02.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:02.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:02.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:02.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:02.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:02.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:02.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:02.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:30:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:30:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:04.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:04.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:04.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:04.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:04.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:04.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:04.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:04.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:04.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:04.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:04.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:04.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:04.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:04.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:04.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:04.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:30:04.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:04.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:30:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:30:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:04.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:30:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:30:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:30:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:30:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:30:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:30:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:04.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:04.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:04.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:30:04.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:04.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:04.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:30:04.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:04.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:04.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:04.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:04.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:30:04.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:04.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:04.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:04.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:04.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:04.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:04.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=79, entry_tid=0' 2026-02-18T06:30:04.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:30:04.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:04.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:04.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:04.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:30:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:30:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:04.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:04.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:30:04.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:04.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:04.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:04.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:04.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:04.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:04.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:04.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=79, entry_tid=0' 2026-02-18T06:30:04.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=79, entry_tid=0' -a 'object_number=0, tag_tid=79, entry_tid=0' = 'object_number=0, tag_tid=79, entry_tid=0' 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:04.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:05.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:30:05.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:05.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:05.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:02 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2026-02-18T06:30:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:02' 2026-02-18T06:30:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:30:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:30:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:30:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:02' 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:30:02' 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:02' 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:05.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:30:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}2026-02-18 06:30:02' 2026-02-18T06:30:05.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:05.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:05.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:02' 2026-02-18T06:30:05.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:30:05.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:05.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:30:05.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:05.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:02 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:02' 2026-02-18T06:30:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:30:06.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:30:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2026-02-18T06:30:06.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:06.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:06.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:06.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:02' 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:30:02' 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:02' 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:06.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:06.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}45714576mirror.0trial1302026-02-18 06:30:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:30:02' 2026-02-18T06:30:06.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:06.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:06.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:02' 2026-02-18T06:30:06.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:06.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:30:06.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:06.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:30:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:30:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:30:06.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:30:06.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:30:06.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:06.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:06.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:06.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:30:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:07.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:07.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:07.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:07.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:07.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:07.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:07.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:07.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:07.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:30:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:09.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:30:09.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:09.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:09.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:09.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:09.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:09.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:09.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:09.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:09.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:09.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:09.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:11.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:07 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:08' 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:11.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:11.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:11.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:11.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:11.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:11.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:07' 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:30:07' 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:07' 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:30:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedstopped2026-02-18 06:30:08' 2026-02-18T06:30:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:07' 2026-02-18T06:30:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:11.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:11.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:11.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:11.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:11.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:12.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:08 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:07' 2026-02-18T06:30:12.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:30:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:12.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:30:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:14.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:14.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:12 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:12' 2026-02-18T06:30:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:14.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:14.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:14.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:14.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:14.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:14.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:12' 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:30:12' 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:12' 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:30:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:30:12' 2026-02-18T06:30:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:14.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:14.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:12' 2026-02-18T06:30:14.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:14.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:14.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:14.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:14.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:14.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:14.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:14.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:30:14.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:14.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:14.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:14.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:30:14.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:30:14.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:30:14.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:30:14.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:14.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:14.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:14.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:30:14.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:30:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:14.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:15.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:15.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:15.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:15.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:15.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:15.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:30:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:17.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:30:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:17.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:17.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:17.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:17.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:17.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:17.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:17.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:17.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:17.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:17.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:17.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:30:17.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:17.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:30:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:30:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:21.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:21.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:21.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:21.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:21.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:21.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:21.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:21.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:21.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:21.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:21.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:21.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:30:21.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:30:21.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:30:21.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:30:21.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:30:21.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:21.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:30:21.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:21.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:30:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:30:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:30:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:30:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:21.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:21.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:21.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:21.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:21.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:30:21.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:21.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:30:21.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:21.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:21.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:21.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:30:21.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:21.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:21.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:21.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:21.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:21.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:21.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:21.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=83, entry_tid=0' 2026-02-18T06:30:21.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:30:21.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:21.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:21.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:21.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:30:21.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:21.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:21.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:21.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:30:21.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:21.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:21.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:21.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:21.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:30:21.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:21.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:21.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=83, entry_tid=0' 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=83, entry_tid=0' -a 'object_number=0, tag_tid=83, entry_tid=0' = 'object_number=0, tag_tid=83, entry_tid=0' 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:21.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:21.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:22.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:22.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:22 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:22' 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:30:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:30:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:22.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:22.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:22' 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:30:22' 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:22' 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:22.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:30:22cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}2026-02-18 06:30:22' 2026-02-18T06:30:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:22.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:22' 2026-02-18T06:30:22.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:22.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:30:22.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:22.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:30:22.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:30:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:23.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:23.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:23.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:23.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:23.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2026-02-18T06:30:23.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:23.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:22 2026-02-18T06:30:23.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:23.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:23.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:23.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:23.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:22' 2026-02-18T06:30:23.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:30:23.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:30:23.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:23.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2026-02-18T06:30:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:23.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:23.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:22' 2026-02-18T06:30:23.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:30:22' 2026-02-18T06:30:23.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:22' 2026-02-18T06:30:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:23.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:23.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}43444348mirror.0trial1302026-02-18 06:30:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:30:22' 2026-02-18T06:30:23.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:23.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:23.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:22' 2026-02-18T06:30:23.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:23.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:30:23.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:23.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:30:23.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:30:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:30:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:30:23.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:30:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:30:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:30:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:24.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:24.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:24.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:24.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:24.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:25.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:25.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:25.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:25.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:25.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:25.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:25.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:25.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:30:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:27.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:27.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:27.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:27.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:30:27.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:27.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:27.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:28.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:28.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:27 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:27' 2026-02-18T06:30:28.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:27' 2026-02-18T06:30:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:30:27' 2026-02-18T06:30:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:27' 2026-02-18T06:30:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:28.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:30:27cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:30:27' 2026-02-18T06:30:28.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:28.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:28.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:27' 2026-02-18T06:30:28.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:28.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:28.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:28.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:28.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:29.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:30:29.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:29.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:27 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:27' 2026-02-18T06:30:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:29.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:29.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:29.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:29.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:29.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:29.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:29.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:29.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:29.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:27' 2026-02-18T06:30:29.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:30:27' 2026-02-18T06:30:29.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:29.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:29.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:29.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:29.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:27' 2026-02-18T06:30:29.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:29.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:29.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:29.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:30:27cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:30:27' 2026-02-18T06:30:29.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:29.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:29.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:27' 2026-02-18T06:30:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:29.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:29.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:29.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:29.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:30:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:29.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:30:29.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:30:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:30:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:30:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:30.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:30.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:30.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:30.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:30.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:30.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:30.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:30.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:30.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:30.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:30.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:30.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:30.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:30:30.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:30.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:32.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:30:32.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:32.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:32.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:32.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:32.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:32.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:32.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:32.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:32.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:32.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:32.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:32.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:30:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:32.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:30:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:30:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:36.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:36.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:36.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:36.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:36.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:36.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:36.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:36.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:36.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:36.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:36.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:36.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:36.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:30:36.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:36.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:36.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:36.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:30:36.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:30:36.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:30:36.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:30:36.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:30:36.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:30:36.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:36.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:36.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:36.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:30:36.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:36.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:30:36.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:36.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:30:36.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:30:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:30:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:36.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:36.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:36.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:36.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:30:36.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:30:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:36.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:36.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:36.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:36.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:30:36.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:36.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:36.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=84, entry_tid=0' 2026-02-18T06:30:36.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:30:36.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:36.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:36.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:36.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:36.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:30:36.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:36.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:36.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:36.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:36.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:36.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:30:36.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=84, entry_tid=0' 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=84, entry_tid=0' -a 'object_number=0, tag_tid=84, entry_tid=0' = 'object_number=0, tag_tid=84, entry_tid=0' 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:36.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:36.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:37.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:37.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:37.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:37.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:37.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:37 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:32' 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:30:37.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:30:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:37.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:30:37.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:37.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:37.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:37.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:37' 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:30:37' 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:37' 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:37.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:37.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:30:37cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}2026-02-18 06:30:32' 2026-02-18T06:30:37.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:37.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:37.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:37' 2026-02-18T06:30:37.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:37.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:30:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:30:37.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2026-02-18T06:30:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:32 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:37' 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:30:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:38.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2026-02-18T06:30:38.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:38.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:38.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:38.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:38.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:38.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:38.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:38.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:32' 2026-02-18T06:30:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:30:32' 2026-02-18T06:30:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:32' 2026-02-18T06:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:38.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}45714576mirror.0trial1302026-02-18 06:30:32cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:30:37' 2026-02-18T06:30:38.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:38.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:38.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:32' 2026-02-18T06:30:38.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:38.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:30:38.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:38.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:30:38.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:30:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:30:39.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:30:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:30:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:40.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:40.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:40.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:40.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:40.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:40.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:40.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:40.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:40.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:40.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:40.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:40.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:40.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:40.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:40.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:30:40.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:40.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:30:42.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:30:42.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:30:42.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:42.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:42.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:30:42.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:42.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:42.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:42.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:42.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:43.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:43.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:43.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:43.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:43.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:43.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:42 2026-02-18T06:30:43.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:43.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:43.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:43.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:43.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:42' 2026-02-18T06:30:43.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:43.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:43.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:43.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:43.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:43.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:43.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:43.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:43.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:42' 2026-02-18T06:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:30:42' 2026-02-18T06:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:43.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:43.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:42' 2026-02-18T06:30:43.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:43.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:43.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:30:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:30:42' 2026-02-18T06:30:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:43.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:42' 2026-02-18T06:30:43.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:43.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:43.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:43.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:43.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:43.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:43.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:44.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:42 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:44.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:42' 2026-02-18T06:30:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:44.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:44.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:44.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:44.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:44.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:44.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:42' 2026-02-18T06:30:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:30:42' 2026-02-18T06:30:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:44.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:42' 2026-02-18T06:30:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:44.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:44.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:30:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:30:42' 2026-02-18T06:30:44.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:44.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:44.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:42' 2026-02-18T06:30:44.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:44.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:44.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:44.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:44.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:30:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:30:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:30:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:30:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:30:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:30:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:44.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:45.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:45.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:45.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:45.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:45.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:45.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:45.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:45.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:45.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:45.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:45.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:45.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:45.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:45.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:45.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:45.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:30:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:45.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:30:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:47.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:47.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:47.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:47.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:47.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:47.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:47.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:47.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:47.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:47.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:47.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:47.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:47.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:47.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:47.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:47.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:47.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:47.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:30:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:30:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:30:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:30:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:51.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:51.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:51.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:51.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:51.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:51.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:51.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:51.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:51.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:51.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:51.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:51.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:51.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:51.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:51.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:51.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:51.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:51.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:51.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:30:51.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:51.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:30:51.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:30:51.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:30:51.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:30:51.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:30:51.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:51.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:30:51.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:30:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:30:51.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=88, entry_tid=0' 2026-02-18T06:30:52.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:30:52.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:52.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:52.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:30:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:30:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:30:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:30:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:30:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=88, entry_tid=0' 2026-02-18T06:30:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=88, entry_tid=0' -a 'object_number=0, tag_tid=88, entry_tid=0' = 'object_number=0, tag_tid=88, entry_tid=0' 2026-02-18T06:30:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:30:52.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:52.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:52.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:52.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:52.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:52.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:52.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:52.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:52 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:52' 2026-02-18T06:30:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:30:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:30:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:30:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:53.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:53.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:53.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:53.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:53.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:53.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:52' 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:30:52' 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:52' 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:30:52cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}2026-02-18 06:30:52' 2026-02-18T06:30:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:53.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:52' 2026-02-18T06:30:53.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:53.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:30:53.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:30:53.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:30:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:54.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:54.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:30:54.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:52 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:52' 2026-02-18T06:30:54.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:30:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:30:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2026-02-18T06:30:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:54.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:54.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:54.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:52' 2026-02-18T06:30:54.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:30:52' 2026-02-18T06:30:54.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:54.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:54.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:54.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:30:54.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:52' 2026-02-18T06:30:54.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:54.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:54.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}43444348mirror.0trial1302026-02-18 06:30:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:30:52' 2026-02-18T06:30:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:52' 2026-02-18T06:30:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:54.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:30:54.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:30:54.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:30:54.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:30:54.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:30:54.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:30:54.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:54.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:54.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:54.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:30:54.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:54.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:54.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:54.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:30:54.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:54.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:54.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:55.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:55.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:55.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:55.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:55.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:55.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:55.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:30:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:30:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:30:55.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:30:55.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:30:55.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:55.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:30:57.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:30:57.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:57.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:57.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:57.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:30:57.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:30:57.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:30:57.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:30:57.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:30:57.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:57.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:30:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:30:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:30:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:30:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:30:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:30:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:30:57.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:30:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:30:57.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:30:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:30:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:30:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:30:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:30:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:30:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:30:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:30:57.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:57.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:57.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:30:57.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:30:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:30:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:30:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:30:57.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:30:57.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:30:57.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:57.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:30:57.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:57.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:57.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:57 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:58.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:30:58.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:58.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:58.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:57' 2026-02-18T06:30:58.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:58.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:58.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:58.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:58.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:58.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:57' 2026-02-18T06:30:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:30:57' 2026-02-18T06:30:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:57' 2026-02-18T06:30:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:58.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:58.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:30:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:30:57' 2026-02-18T06:30:58.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:58.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:58.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:57' 2026-02-18T06:30:58.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:58.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:58.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:58.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:58.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:58.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:58.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:58.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:30:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:30:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:30:59.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:57 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:30:57' 2026-02-18T06:30:59.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:30:59.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:30:59.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:30:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:30:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:30:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:59.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:30:59.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:59.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:30:59.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:30:59.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:30:59.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:30:59.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:57' 2026-02-18T06:30:59.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:30:57' 2026-02-18T06:30:59.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:59.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:59.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:59.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:30:59.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:30:57' 2026-02-18T06:30:59.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:30:59.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:30:59.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:30:59.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:30:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:30:57' 2026-02-18T06:30:59.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:30:59.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:30:59.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:30:57' 2026-02-18T06:30:59.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:30:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:30:59.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:30:59.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:30:59.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:30:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:30:59.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:30:59.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:30:59.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:30:59.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:00.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:00.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:00.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:00.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:00.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:31:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:31:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:02.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:02.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:02.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:02.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:02.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:02.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:02.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:02.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:02.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:02.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:02.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:02.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:02.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:02.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:02.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:02.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:02.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:02.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:02.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:31:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:31:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:31:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:31:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:31:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:31:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:31:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:02.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:31:03.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:03.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:31:03.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:03.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:03.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:03.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:31:03.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:03.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:03.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:03.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:31:03.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:03.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:03.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:03.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:03.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:31:03.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:03.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:03.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:03.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:03.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:03.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:03.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:03.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=89, entry_tid=0' 2026-02-18T06:31:03.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:31:03.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:03.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:03.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:03.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:03.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:03.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:31:03.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=89, entry_tid=0' 2026-02-18T06:31:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=89, entry_tid=0' -a 'object_number=0, tag_tid=89, entry_tid=0' = 'object_number=0, tag_tid=89, entry_tid=0' 2026-02-18T06:31:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:31:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:03.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:04.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:04.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:02 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:04.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2026-02-18T06:31:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:02' 2026-02-18T06:31:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:31:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:31:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:31:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:04.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:04.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:04.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:04.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:02' 2026-02-18T06:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:31:02' 2026-02-18T06:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:02' 2026-02-18T06:31:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:04.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:31:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}2026-02-18 06:31:02' 2026-02-18T06:31:04.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:04.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:04.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:02' 2026-02-18T06:31:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:31:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:31:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:31:04.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:04.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:05.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:31:05.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:05.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:02 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:02' 2026-02-18T06:31:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:31:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2026-02-18T06:31:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:02' 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:31:02' 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:02' 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:05.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}45714576mirror.0trial1302026-02-18 06:31:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:31:02' 2026-02-18T06:31:05.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:05.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:05.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:02' 2026-02-18T06:31:05.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:05.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:31:05.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:05.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:31:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:31:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:31:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:31:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:06.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:06.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:06.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:06.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:06.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:31:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:08.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:08.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:08.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:08.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:08.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:08.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:08.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:08.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:08.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:08.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:08.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:08.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:08.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:08.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:08.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:08.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:31:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:31:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:09.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:09.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:07 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:07' 2026-02-18T06:31:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:09.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:09.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:09.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:09.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:09.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:07' 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:31:07' 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:07' 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:09.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:31:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:31:07' 2026-02-18T06:31:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:07' 2026-02-18T06:31:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:09.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:09.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:09.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:09.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:09.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:09.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:31:09.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:09.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:31:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:10.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:07 2026-02-18T06:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:07' 2026-02-18T06:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:07' 2026-02-18T06:31:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:31:07' 2026-02-18T06:31:10.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:07' 2026-02-18T06:31:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:31:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:31:07' 2026-02-18T06:31:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:10.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:07' 2026-02-18T06:31:10.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:10.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:10.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:10.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:31:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:31:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:10.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:10.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:31:10.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:10.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:10.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:31:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:11.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:11.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:11.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:11.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:11.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:11.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:11.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:31:11.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:11.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:13.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:31:13.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:13.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:13.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:13.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:13.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:13.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:13.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:13.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:13.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:13.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:13.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:13.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:13.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:13.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:31:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:31:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:31:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:31:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:31:13.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:31:13.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:13.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:31:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:31:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:31:13.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:31:13.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:31:14.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:14.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:31:14.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:14.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:14.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:14.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:31:14.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:14.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:14.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:14.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:31:14.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:14.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:14.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:14.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:14.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:31:14.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:14.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=93, entry_tid=0' 2026-02-18T06:31:14.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:14.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:14.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:14.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:31:14.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=93, entry_tid=0' 2026-02-18T06:31:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=93, entry_tid=0' -a 'object_number=0, tag_tid=93, entry_tid=0' = 'object_number=0, tag_tid=93, entry_tid=0' 2026-02-18T06:31:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:31:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:14.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:14.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:14.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:14.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:14.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:14.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:14.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:31:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:12 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:12' 2026-02-18T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:31:15.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:31:15.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:15.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:31:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:15.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:15.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:15.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:12' 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:31:12' 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:12' 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:15.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:31:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}2026-02-18 06:31:12' 2026-02-18T06:31:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:12' 2026-02-18T06:31:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:15.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:31:15.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:31:15.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:31:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:31:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:16.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:16.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:16.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:16.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2026-02-18T06:31:16.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:16.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:12 2026-02-18T06:31:16.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:12' 2026-02-18T06:31:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:31:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:31:16.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:16.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2026-02-18T06:31:16.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:16.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:16.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:16.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:16.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:16.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:16.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:16.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:12' 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:31:12' 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:12' 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:16.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}43444348mirror.0trial1302026-02-18 06:31:12cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:31:12' 2026-02-18T06:31:16.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:16.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:16.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:12' 2026-02-18T06:31:16.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:16.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:31:16.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:31:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:31:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:31:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:16.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:16.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:31:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:31:16.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:31:16.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:17.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:17.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:17.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:17.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:17.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:17.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:17.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:17.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:17.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:17.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:17.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:17.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:17.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:17.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:17.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:17.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:17.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:17.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:17.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:17.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:17.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:31:17.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:17.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:19.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:19.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:19.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:19.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:19.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:19.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:19.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:19.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:19.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:19.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:19.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:31:19.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:19.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:20.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:20.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:17 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:17' 2026-02-18T06:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:20.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:20.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:20.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:20.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:20.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:20.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:17' 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:31:17' 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:17' 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:20.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:20.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:31:17cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:31:17' 2026-02-18T06:31:20.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:20.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:17' 2026-02-18T06:31:20.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:21.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:17 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:17' 2026-02-18T06:31:21.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:21.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:21.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:21.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:21.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:21.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:21.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:21.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:21.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:21.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:21.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:21.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:17' 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:31:17' 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:17' 2026-02-18T06:31:21.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:31:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:31:17' 2026-02-18T06:31:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:21.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:21.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:17' 2026-02-18T06:31:21.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:31:21.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:31:21.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:31:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:22.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:22.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:22.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:22.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:22.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:22.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:22.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:22.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:22.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:22.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:31:22.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:31:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:31:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:31:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:31:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:31:22.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:22.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:22.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:22.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:22.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:22.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:31:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:31:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:31:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:22.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:22.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:22.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:22.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:22.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:22.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:22.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=94, entry_tid=0' 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:31:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:31:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:23.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:23.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:23.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=94, entry_tid=0' 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=94, entry_tid=0' -a 'object_number=0, tag_tid=94, entry_tid=0' = 'object_number=0, tag_tid=94, entry_tid=0' 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:23.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:31:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:24.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:24.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:24.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:24.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:24.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:24.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:24.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:22 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:24.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2026-02-18T06:31:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:22' 2026-02-18T06:31:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:31:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:31:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:24.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:31:24.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:24.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:24.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:24.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:24.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:22' 2026-02-18T06:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:31:22' 2026-02-18T06:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:22' 2026-02-18T06:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:24.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:31:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}2026-02-18 06:31:22' 2026-02-18T06:31:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:24.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:22' 2026-02-18T06:31:24.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:31:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:31:24.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:31:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:31:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:25.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:25.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:22 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:22' 2026-02-18T06:31:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:31:25.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:31:25.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:25.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2026-02-18T06:31:25.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:25.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:25.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:25.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:25.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:25.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:22' 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:31:22' 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:22' 2026-02-18T06:31:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:25.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}45714576mirror.0trial1302026-02-18 06:31:22cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:31:22' 2026-02-18T06:31:25.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:25.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:25.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:22' 2026-02-18T06:31:25.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:31:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:31:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:31:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:31:25.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:31:25.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:25.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:25.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:26.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:26.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:26.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:26.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:26.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:26.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:26.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:26.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:26.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:26.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:26.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:26.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:26.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:26.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:26.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:26.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:26.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:26.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:26.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:26.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:31:26.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:26.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:28.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:28.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:28.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:28.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:28.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:28.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:28.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:28.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:28.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:28.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:28.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:28.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:28.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:28.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:28.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:31:28.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:28.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:28.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:29.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:31:29.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:29.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:29.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:27 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:27' 2026-02-18T06:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:27' 2026-02-18T06:31:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:31:27' 2026-02-18T06:31:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:29.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:27' 2026-02-18T06:31:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:31:27cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:31:27' 2026-02-18T06:31:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:29.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:27' 2026-02-18T06:31:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:29.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:29.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:29.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:29.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:29.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:29.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:31:30.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:27 2026-02-18T06:31:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:27' 2026-02-18T06:31:30.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:30.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:30.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:30.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:30.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:30.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:30.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:30.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:30.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:30.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:30.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:30.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:27' 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:31:27' 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:27' 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:31:27cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:31:27' 2026-02-18T06:31:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:30.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:27' 2026-02-18T06:31:30.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:30.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:30.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:30.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:30.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:30.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:30.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:30.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:31:30.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:30.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:30.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:30.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:31:30.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:31:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:31:30.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:31.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:31:31.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:31.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:31.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:31.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:31.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:31.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:31.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:31.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:31.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:31.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:31.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:31.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:31.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:31.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:31.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:31.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:31.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:31.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:31.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:31.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:31.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:31.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:31.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:31.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:31.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:31:31.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:31.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:33.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:31:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:33.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:33.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:33.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:33.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:33.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:33.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:33.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:33.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:33.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:33.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:33.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:33.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:33.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:33.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:33.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:33.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:33.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:33.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:33.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:33.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:31:33.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:31:33.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:31:34.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:34.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:31:34.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:34.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:31:34.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:31:34.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:34.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:31:34.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:34.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:34.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:34.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:34.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:34.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:34.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:34.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:31:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:31:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:31:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=98, entry_tid=0' 2026-02-18T06:31:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:31:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:34.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:34.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:34.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:34.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:34.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:34.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=98, entry_tid=0' 2026-02-18T06:31:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=98, entry_tid=0' -a 'object_number=0, tag_tid=98, entry_tid=0' = 'object_number=0, tag_tid=98, entry_tid=0' 2026-02-18T06:31:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:35.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:35.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:32 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:32' 2026-02-18T06:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:31:35.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:31:35.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:31:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:35.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:35.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:35.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:35.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:35.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:32' 2026-02-18T06:31:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:31:32' 2026-02-18T06:31:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:32' 2026-02-18T06:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:35.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:31:32cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}2026-02-18 06:31:32' 2026-02-18T06:31:35.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:35.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:35.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:32' 2026-02-18T06:31:35.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:35.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:31:35.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:35.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:31:35.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:31:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:36.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:32 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:32' 2026-02-18T06:31:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:31:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:31:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2026-02-18T06:31:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:36.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:36.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:32' 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:31:32' 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:32' 2026-02-18T06:31:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}43444348mirror.0trial1302026-02-18 06:31:32cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:31:32' 2026-02-18T06:31:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:36.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:32' 2026-02-18T06:31:36.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:36.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:31:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:31:36.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:36.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:36.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:36.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:31:36.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:31:36.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:36.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:36.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:36.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:31:36.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:31:36.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:31:36.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:36.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:37.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:37.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:37.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:37.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:37.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:37.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:37.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:37.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:37.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:37.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:31:37.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:37.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:39.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:39.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:39.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:39.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:39.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:39.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:39.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:39.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:39.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:39.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:31:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:31:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:40.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:40.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:37 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:37' 2026-02-18T06:31:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:40.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:40.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:40.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:40.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:40.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:40.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:40.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:40.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:40.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:37' 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:31:37' 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:37' 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:40.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:31:37cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:31:37' 2026-02-18T06:31:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:40.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:40.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:37' 2026-02-18T06:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:40.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:40.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:40.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:41.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:31:41.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:41.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:41.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:41.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:41.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:41.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:41.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:41.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:41.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:37 2026-02-18T06:31:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:37' 2026-02-18T06:31:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:41.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:37' 2026-02-18T06:31:41.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:31:37' 2026-02-18T06:31:41.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:41.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:41.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:41.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:41.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:37' 2026-02-18T06:31:41.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:41.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:41.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:31:37cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:31:37' 2026-02-18T06:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:37' 2026-02-18T06:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:41.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:41.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:41.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:41.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:41.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:31:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:31:41.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:31:41.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:41.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:31:41.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:41.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:41.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:42.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:42.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:42.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:42.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:42.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:42.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:42.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:42.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:42.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:42.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:42.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:42.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:42.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:42.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:42.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:42.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:42.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:42.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:42.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:42.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:42.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:42.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:42.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:42.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:42.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:31:42.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:42.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:31:42.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:31:42.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:42.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:31:42.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:42.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:31:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:31:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:31:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:42.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:42.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:31:42.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:42.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:42.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:43.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:43.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:43.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:43.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:31:43.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:31:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:31:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:43.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:31:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:43.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:31:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:43.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:43.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:43.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:43.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:43.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:43.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:43.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=99, entry_tid=0' 2026-02-18T06:31:43.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:31:43.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:43.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:43.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:43.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:31:43.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=99, entry_tid=0' 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=99, entry_tid=0' -a 'object_number=0, tag_tid=99, entry_tid=0' = 'object_number=0, tag_tid=99, entry_tid=0' 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:31:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:42 2026-02-18T06:31:44.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2026-02-18T06:31:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:42' 2026-02-18T06:31:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:31:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:31:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:42' 2026-02-18T06:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:31:42' 2026-02-18T06:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:42' 2026-02-18T06:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:44.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:31:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}2026-02-18 06:31:42' 2026-02-18T06:31:44.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:44.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:44.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:42' 2026-02-18T06:31:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:44.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:31:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:31:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:44.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:44.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:45.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:31:45.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:45.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:45.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:45.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:42 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:42' 2026-02-18T06:31:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:31:45.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:31:45.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:45.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2026-02-18T06:31:45.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:45.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:45.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:45.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:45.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:45.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:45.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:45.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:45.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:42' 2026-02-18T06:31:45.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:31:42' 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:42' 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:45.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:45.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}45714576mirror.0trial1302026-02-18 06:31:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:31:42' 2026-02-18T06:31:45.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:45.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:45.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:42' 2026-02-18T06:31:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:31:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:31:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:31:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:45.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:45.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:45.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:31:45.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:31:45.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:45.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:45.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:46.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:46.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:46.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:46.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:46.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:46.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:46.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:31:46.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:48.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:31:48.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:48.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:48.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:48.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:48.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:48.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:48.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:31:48.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:31:48.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:31:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:31:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:31:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:31:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:31:48.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:48.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:48.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:31:48.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:48.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:48.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:48.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:48.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:48.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:31:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:49.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:49.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:49.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:49.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:49.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:49.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:49.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:47 2026-02-18T06:31:49.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:49.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:49.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:49.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:49.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:47' 2026-02-18T06:31:49.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:49.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:49.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:49.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:49.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:47' 2026-02-18T06:31:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:31:47' 2026-02-18T06:31:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:47' 2026-02-18T06:31:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:31:47cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:31:47' 2026-02-18T06:31:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:49.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:47' 2026-02-18T06:31:49.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:49.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:49.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:49.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:49.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:31:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:49.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:49.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:49.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:49.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:49.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:50.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:50.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:47 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:47' 2026-02-18T06:31:50.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:31:50.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:31:50.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:50.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:31:50.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:50.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:50.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:50.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:50.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:50.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:50.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:50.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:47' 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:31:47' 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:47' 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:50.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:50.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:50.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:31:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:31:47' 2026-02-18T06:31:50.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:50.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:50.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:47' 2026-02-18T06:31:50.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:50.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:31:50.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:31:50.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:31:50.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:31:50.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:31:50.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:31:50.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:50.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:51.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:31:51.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:51.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:51.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:51.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:51.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:51.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:51.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:51.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:51.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:51.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:51.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:51.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:51.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:51.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:51.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:31:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:53.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:53.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:53.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:53.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:53.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:53.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:53.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:53.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:53.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:53.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:53.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:53.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:31:53.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:31:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:31:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:31:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:31:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:31:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:53.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:31:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:31:53.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:53.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:31:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:31:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:53.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:31:54.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:31:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:54.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:31:54.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:54.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:54.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:54.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=103, entry_tid=0' 2026-02-18T06:31:54.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:31:54.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:31:54.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:31:54.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:54.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:31:54.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:31:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:31:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:31:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:31:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:31:54.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=103, entry_tid=0' 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=103, entry_tid=0' -a 'object_number=0, tag_tid=103, entry_tid=0' = 'object_number=0, tag_tid=103, entry_tid=0' 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:54.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:55.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:55.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:31:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:52 2026-02-18T06:31:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:31:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2026-02-18T06:31:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:52' 2026-02-18T06:31:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:31:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:31:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:31:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:55.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:55.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:55.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:55.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:52' 2026-02-18T06:31:55.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:31:52' 2026-02-18T06:31:55.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:55.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:55.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:55.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:31:55.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:52' 2026-02-18T06:31:55.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:55.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:55.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:31:52cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}2026-02-18 06:31:52' 2026-02-18T06:31:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:52' 2026-02-18T06:31:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:31:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:31:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:31:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:55.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:55.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:55.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:31:56.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:31:56.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:31:56.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:52 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:52' 2026-02-18T06:31:56.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:31:56.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:31:56.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:31:56.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2026-02-18T06:31:56.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:31:56.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:56.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:31:56.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:56.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:31:56.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:31:56.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:31:56.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:52' 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:31:52' 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:52' 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:31:56.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:31:56.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}43444348mirror.0trial1302026-02-18 06:31:52cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:31:52' 2026-02-18T06:31:56.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:31:56.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:31:56.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:52' 2026-02-18T06:31:56.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:31:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:31:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:31:56.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:31:56.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:31:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:31:56.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:31:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:31:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:56.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:31:56.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:31:56.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:56.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:31:57.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:57.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:57.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:57.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:57.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:57.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:57.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:57.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:57.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:57.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:57.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:57.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:31:57.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:31:57.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:57.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:31:57.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:31:57.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:31:57.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:57.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:31:59.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:31:59.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:31:59.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:31:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:31:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:31:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:31:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:31:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:31:59.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:31:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:31:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:31:59.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:31:59.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:31:59.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:31:59.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:31:59.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:31:59.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:00.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:32:00.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:00.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:00.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:00.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:57 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:00.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:57' 2026-02-18T06:32:00.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:00.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:00.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:00.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:00.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:00.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:00.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:00.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:00.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:57' 2026-02-18T06:32:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:31:57' 2026-02-18T06:32:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:57' 2026-02-18T06:32:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:00.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:31:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:31:57' 2026-02-18T06:32:00.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:57' 2026-02-18T06:32:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:00.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:00.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:00.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:00.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:32:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:00.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:00.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:01.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:32:01.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:57 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:31:57' 2026-02-18T06:32:01.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:01.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:01.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:01.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:01.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:01.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:01.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:01.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:57' 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:31:57' 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:31:57' 2026-02-18T06:32:01.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:31:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:31:57' 2026-02-18T06:32:01.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:01.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:01.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:31:57' 2026-02-18T06:32:01.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:01.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:01.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:01.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:01.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:01.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:01.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:01.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:32:01.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:01.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:01.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:01.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:32:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:32:01.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:32:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:32:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:32:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:01.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:01.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:01.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:32:01.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:01.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:01.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:02.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:02.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:02.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:02.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:02.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:02.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:02.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:02.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:02.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:02.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:02.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:02.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:32:02.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:02.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:32:02.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:32:02.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:32:02.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:32:02.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:32:02.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:32:02.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:32:02.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:02.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:32:02.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:02.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:02.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:02.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:32:02.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:32:02.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:02.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:02.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:02.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:02.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:02.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:02.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:32:02.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:32:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:32:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:32:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:02.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:32:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:02.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:02.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:02.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:02.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:02.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:02.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:02.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=104, entry_tid=0' 2026-02-18T06:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:32:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:32:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:03.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:03.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=104, entry_tid=0' 2026-02-18T06:32:03.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=104, entry_tid=0' -a 'object_number=0, tag_tid=104, entry_tid=0' = 'object_number=0, tag_tid=104, entry_tid=0' 2026-02-18T06:32:03.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:03.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:04.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:02 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:02' 2026-02-18T06:32:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:32:04.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:32:04.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:04.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:32:04.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:04.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:04.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:04.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:04.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:04.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:04.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:04.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:02' 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:32:02' 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:02' 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:04.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:04.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:32:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}2026-02-18 06:32:02' 2026-02-18T06:32:04.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:04.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:04.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:02' 2026-02-18T06:32:04.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:04.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:32:04.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:04.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:32:04.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:04.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:04.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:04.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:32:04.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:04.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:04.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:04.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:04.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:04.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:04.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:04.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:05.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:05.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:05.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:05.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:02 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:02' 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:32:05.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:05.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2026-02-18T06:32:05.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:05.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:05.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:05.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:05.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:05.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:05.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:05.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:05.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:02' 2026-02-18T06:32:05.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:32:02' 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:02' 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}45714576mirror.0trial1302026-02-18 06:32:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:32:02' 2026-02-18T06:32:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:02' 2026-02-18T06:32:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:32:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:32:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:32:05.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:05.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:05.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:06.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:06.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:06.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:06.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:06.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:06.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:06.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:06.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:06.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:06.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:06.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:06.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:06.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:06.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:06.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:06.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:32:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:06.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:08.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:08.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:08.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:08.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:08.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:08.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:08.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:08.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:08.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:08.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:08.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:08.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:08.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:08.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:08.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:07 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:07' 2026-02-18T06:32:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:09.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:09.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:09.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:09.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:07' 2026-02-18T06:32:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:32:07' 2026-02-18T06:32:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:09.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:09.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:09.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:09.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:07' 2026-02-18T06:32:09.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:09.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:09.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:09.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:32:07cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:32:07' 2026-02-18T06:32:09.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:09.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:09.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:07' 2026-02-18T06:32:09.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:09.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:09.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:09.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:09.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:09.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:07 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:07' 2026-02-18T06:32:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:10.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:10.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:10.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:10.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:10.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:10.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:10.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:07' 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:32:07' 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:07' 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:10.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:32:07cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:32:07' 2026-02-18T06:32:10.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:10.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:10.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:07' 2026-02-18T06:32:10.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:10.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:10.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:10.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:10.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:10.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:32:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:32:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:32:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:32:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:10.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:11.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:11.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:11.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:11.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:11.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:11.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:32:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:13.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:13.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:13.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:13.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:13.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:13.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:13.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:13.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:13.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:13.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:13.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:32:13.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:32:13.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:32:13.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:32:13.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:32:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:32:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:32:14.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:32:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:14.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:32:14.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:14.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:32:14.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:14.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:14.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:32:14.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:32:14.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:14.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:14.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:14.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:14.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:14.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:14.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:32:14.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:14.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:32:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:32:14.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:14.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:14.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=108, entry_tid=0' 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:32:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:32:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=108, entry_tid=0' 2026-02-18T06:32:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=108, entry_tid=0' -a 'object_number=0, tag_tid=108, entry_tid=0' = 'object_number=0, tag_tid=108, entry_tid=0' 2026-02-18T06:32:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:14.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:15.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:12 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2026-02-18T06:32:15.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:12' 2026-02-18T06:32:15.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:32:15.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:32:15.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:15.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:32:15.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:15.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:15.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:15.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:15.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:15.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:15.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:15.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:12' 2026-02-18T06:32:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:32:12' 2026-02-18T06:32:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:12' 2026-02-18T06:32:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:15.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:15.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:32:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}2026-02-18 06:32:12' 2026-02-18T06:32:15.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:15.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:15.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:12' 2026-02-18T06:32:15.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:15.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:32:15.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:15.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:32:15.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:15.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:15.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:15.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:32:15.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:15.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:15.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:15.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:15.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:15.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:15.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:15.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:16.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:32:16.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:16.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:16.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:12 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:16.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:16.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:16.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:12' 2026-02-18T06:32:16.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:32:16.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:32:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2026-02-18T06:32:16.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:16.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:16.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:16.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:16.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:16.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:12' 2026-02-18T06:32:16.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:32:12' 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:12' 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:16.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}43444348mirror.0trial1302026-02-18 06:32:12cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:32:12' 2026-02-18T06:32:16.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:16.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:16.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:12' 2026-02-18T06:32:16.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:16.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:32:16.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:16.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:32:16.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:16.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:16.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:16.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:32:16.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:32:16.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:16.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:16.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:16.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:32:16.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:32:16.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:32:16.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:16.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:16.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:16.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:32:16.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:16.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:32:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:16.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:17.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:32:17.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:17.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:17.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:17.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:17.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:17.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:17.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:17.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:17.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:17.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:17.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:17.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:17.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:17.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:17.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:17.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:17.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:17.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:17.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:32:17.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:17.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:32:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:19.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:19.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:19.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:19.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:19.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:19.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:19.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:19.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:19.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:19.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:19.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:19.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:19.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:19.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:19.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:19.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:19.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:19.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:19.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:32:19.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:19.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:32:19.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:19.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:19.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:19.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:19.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:19.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:19.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:19.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:20.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:20.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:20.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:20.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:20.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:20.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:20.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:20.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:17 2026-02-18T06:32:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:17' 2026-02-18T06:32:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:20.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:20.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:20.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:20.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:20.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:20.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:17' 2026-02-18T06:32:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:32:17' 2026-02-18T06:32:20.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:17' 2026-02-18T06:32:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:20.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:32:17cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:32:17' 2026-02-18T06:32:20.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:20.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:17' 2026-02-18T06:32:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:20.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:20.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:20.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:20.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:21.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:17 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:17' 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:21.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:21.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:21.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:21.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:21.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:21.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:21.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:21.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:17' 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:32:17' 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:17' 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:21.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:21.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:32:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:32:17' 2026-02-18T06:32:21.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:21.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:21.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:17' 2026-02-18T06:32:21.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:21.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:21.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:21.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:21.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:21.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:21.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:21.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:32:21.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:21.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:21.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:21.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:32:21.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:32:21.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:32:21.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:32:21.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:21.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:21.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:21.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:32:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:32:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:32:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:22.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:22.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:22.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:22.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:22.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:22.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:22.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:22.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:22.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:22.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:22.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:22.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:32:22.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:32:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:22.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:32:22.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:22.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:32:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:32:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:23.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:32:23.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:23.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:23.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:23.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:23.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:32:23.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:23.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=109, entry_tid=0' 2026-02-18T06:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:23.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:23.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:32:23.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:23.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:23.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:23.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:23.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:23.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:23.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=109, entry_tid=0' 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=109, entry_tid=0' -a 'object_number=0, tag_tid=109, entry_tid=0' = 'object_number=0, tag_tid=109, entry_tid=0' 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:23.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:24.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:24.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:24.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:22 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:22' 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:32:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:24.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:24.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:22' 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:32:22' 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:22' 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:24.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:32:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}2026-02-18 06:32:22' 2026-02-18T06:32:24.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:24.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:24.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:22' 2026-02-18T06:32:24.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:24.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:32:24.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:24.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:32:24.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:24.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:24.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:24.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:32:24.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:24.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:24.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:25.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:22 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:22' 2026-02-18T06:32:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:32:25.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:32:25.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:25.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2026-02-18T06:32:25.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:25.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:25.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:25.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:25.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:22' 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:32:22' 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:22' 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:25.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}45714576mirror.0trial1302026-02-18 06:32:22cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:32:22' 2026-02-18T06:32:25.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:25.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:25.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:22' 2026-02-18T06:32:25.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:25.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:32:25.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:25.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:32:25.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:32:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:25.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:32:25.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:32:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:26.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:26.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:26.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:26.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:26.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:26.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:26.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:26.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:26.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:26.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:26.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:26.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:26.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:26.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:32:26.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:26.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:28.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:28.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:28.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:28.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:28.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:28.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:28.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:28.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:28.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:28.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:28.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:28.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:28.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:28.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:28.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:28.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:28.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:32:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:32:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:28.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:28.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:28.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:28.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:28.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:29.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:27 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:27' 2026-02-18T06:32:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:27' 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:32:27' 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:27' 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:32:27cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:32:27' 2026-02-18T06:32:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:29.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:29.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:27' 2026-02-18T06:32:29.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:30.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:30.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:27 2026-02-18T06:32:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:27' 2026-02-18T06:32:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:30.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:30.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:30.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:30.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:30.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:30.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:30.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:30.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:30.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:30.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:30.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:30.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:27' 2026-02-18T06:32:30.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:32:27' 2026-02-18T06:32:30.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:30.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:30.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:30.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:30.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:27' 2026-02-18T06:32:30.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:30.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:30.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:30.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:32:27cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:32:27' 2026-02-18T06:32:30.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:30.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:30.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:27' 2026-02-18T06:32:30.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:30.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:30.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:30.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:30.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:32:30.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:32:30.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:32:30.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:30.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:31.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:31.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:31.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:31.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:31.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:31.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:31.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:31.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:31.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:31.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:31.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:31.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:31.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:31.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:32:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:31.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:33.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:32:33.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:33.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:33.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:33.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:33.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:33.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:33.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:33.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:33.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:33.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:33.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:33.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:33.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:33.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:32:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:32:33.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:32:33.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:32:33.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:33.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:32:33.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:33.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:32:33.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:32:33.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:33.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:33.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:34.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:32:34.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:34.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:32:34.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:34.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:34.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:34.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:32:34.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:32:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:34.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:34.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:34.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:32:34.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:34.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:34.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:34.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:34.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:34.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:34.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:34.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=113, entry_tid=0' 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:34.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:34.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:32:34.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:34.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:34.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:34.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:34.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:34.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:32:34.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:34.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=113, entry_tid=0' 2026-02-18T06:32:34.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=113, entry_tid=0' -a 'object_number=0, tag_tid=113, entry_tid=0' = 'object_number=0, tag_tid=113, entry_tid=0' 2026-02-18T06:32:34.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:34.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:32:34.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:34.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:34.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:34.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:35.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:32 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:32' 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:32:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:32:35.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:35.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:32:35.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:35.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:35.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:35.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:35.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:35.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:32' 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:32:32' 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:32' 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:35.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:32:32cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}2026-02-18 06:32:32' 2026-02-18T06:32:35.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:35.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:32' 2026-02-18T06:32:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:35.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:32:35.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:35.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:32:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:32:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:36.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:32:36.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:36.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:32 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:36.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:36.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:32' 2026-02-18T06:32:36.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:32:36.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:32:36.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:36.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2026-02-18T06:32:36.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:36.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:36.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:36.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:36.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:32' 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:32:32' 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:32' 2026-02-18T06:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:36.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}43444348mirror.0trial1302026-02-18 06:32:32cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:32:32' 2026-02-18T06:32:36.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:36.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:32' 2026-02-18T06:32:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:32:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:32:36.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:32:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test 2026-02-18T06:32:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-18T06:32:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:32:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:32:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:37.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:32:37.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:37.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:37.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:37.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:37.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:37.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:37.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:37.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:37.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:37.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:37.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:37.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:37.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:37.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:37.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:37.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:37.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:37.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:37.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:37.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:37.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:32:37.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:37.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:39.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:39.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:39.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:39.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:39.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:39.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:39.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:39.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:39.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:32:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:39.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:40.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:40.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:40.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:37 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:37' 2026-02-18T06:32:40.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:32:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:32:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:42.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:42.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:42.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:42.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:42.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:42.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:42.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:42 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:37' 2026-02-18T06:32:42.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:42.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:42.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:42.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:42.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:42.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:42.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:42.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:42.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:42.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:42.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:42.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:42.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:42' 2026-02-18T06:32:42.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:32:42' 2026-02-18T06:32:42.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:42.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:42.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:42.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:42.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:42' 2026-02-18T06:32:42.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:42.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:42.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:42.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:32:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:32:42' 2026-02-18T06:32:42.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:42.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:42.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:42' 2026-02-18T06:32:42.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:42.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:42.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:42.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:43.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:32:43.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:43.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:42 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:42' 2026-02-18T06:32:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:43.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:43.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:43.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:43.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:43.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:42' 2026-02-18T06:32:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:32:42' 2026-02-18T06:32:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:43.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:42' 2026-02-18T06:32:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:43.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:43.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:32:42cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:32:42' 2026-02-18T06:32:43.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:43.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:43.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:42' 2026-02-18T06:32:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:43.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:43.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:43.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:43.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test 2026-02-18T06:32:43.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:43.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:43.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:43.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:32:43.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-18T06:32:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:32:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-18T06:32:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:43.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-18T06:32:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:32:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:43.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:44.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:44.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:44.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:44.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:44.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:44.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:44.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:44.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:44.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:32:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:32:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:46.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:46.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:46.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:32:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-18T06:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:50.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:50.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:50.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:50.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:50.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:50.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:50.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:32:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:50.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror mirror test 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster2 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster2 mirror test 2026-02-18T06:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:51.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:32:51.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:32:51.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:51.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:51.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:51.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:51.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:32:51.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster1 mirror test 2026-02-18T06:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '' 2026-02-18T06:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:51.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:51.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:51.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:51.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:51.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:51.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=114, entry_tid=0' 2026-02-18T06:32:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster1 mirror test 2026-02-18T06:32:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:51.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster1 mirror test '..*' 2026-02-18T06:32:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:32:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:32:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:32:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:32:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:32:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:32:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2026-02-18T06:32:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:32:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:32:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:32:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=ee4dbd21-9013-44f3-9ef7-aff3b2fab932, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2026-02-18T06:32:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:32:51.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=114, entry_tid=0' 2026-02-18T06:32:51.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=114, entry_tid=0' -a 'object_number=0, tag_tid=114, entry_tid=0' = 'object_number=0, tag_tid=114, entry_tid=0' 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:52.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-18T06:32:52.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:52.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:52.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:52.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:52.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:52.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:52.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:52.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:52.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:52.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:47 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:47' 2026-02-18T06:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:32:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:32:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:32:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:52.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:52.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:52.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:52.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:52.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:47' 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-18 06:32:47' 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:47' 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:52.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary43444348mirror.0trial1302026-02-18 06:32:47cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}2026-02-18 06:32:47' 2026-02-18T06:32:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:52.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:47' 2026-02-18T06:32:52.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:52.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:32:52.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:32:52.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:32:52.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:32:52.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-18T06:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:53.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:53.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:47 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:32:53.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:32:53.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:52' 2026-02-18T06:32:53.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:32:53.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:32:53.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:53.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2026-02-18T06:32:53.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:53.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:53.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:53.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:53.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:53.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:47' 2026-02-18T06:32:53.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-18 06:32:47' 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:47' 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}45714576mirror.0trial1302026-02-18 06:32:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:32:52' 2026-02-18T06:32:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:53.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:47' 2026-02-18T06:32:53.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:53.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:32:53.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:32:53.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:32:53.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:32:53.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:53.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror test 2026-02-18T06:32:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:53.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-18T06:32:53.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:32:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-18T06:32:53.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:53.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:54.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:54.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:54.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:54.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:54.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:54.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:54.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:54.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:54.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:54.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:54.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:54.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:54.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:32:54.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:32:54.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:54.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:32:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:32:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = started 2026-02-18T06:32:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:54.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:32:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-18T06:32:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:56.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:32:56.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:56.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:56.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-18T06:32:56.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:56.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:32:56.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-18T06:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:32:56.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:32:56.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:56.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:56.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:32:56.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:56.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:56.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:32:56.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:56.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:56.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:56.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:56.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:56.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:57.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-18T06:32:57.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:57.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:57.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:57.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:57.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:32:57.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:57.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:57.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:57.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:57.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:57.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:57 2026-02-18T06:32:57.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:57.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:32:57.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:57.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:57.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:57' 2026-02-18T06:32:57.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:57.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:57.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:57.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:57.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:57.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:57.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:57.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:57.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:57.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:57.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:57.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:57.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:57' 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-18 06:32:57' 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:57' 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:57.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:57.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:57.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary43444348mirror.0trial1302026-02-18 06:32:57cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:32:57' 2026-02-18T06:32:57.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:57.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:57.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:57' 2026-02-18T06:32:57.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:57.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:57.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:57.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:57.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-18T06:32:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:57.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:57.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:58.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-18T06:32:58.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:58.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:58.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:58.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:58.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:32:58.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:32:58.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:58.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:58.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:57 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:32:58.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:32:58.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:32:58.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:32:57' 2026-02-18T06:32:58.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:32:58.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:32:58.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:32:58.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:32:58.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:32:58.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:58.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:32:58.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:58.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:32:58.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:32:58.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:32:58.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:57' 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-18 06:32:57' 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:32:57' 2026-02-18T06:32:58.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:32:58.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:32:58.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:32:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+unknownremote image is not primary2026-02-18 06:32:57' 2026-02-18T06:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:32:58.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:32:57' 2026-02-18T06:32:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:32:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:32:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:32:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:32:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:32:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:32:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster2 mirror test 2026-02-18T06:32:58.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:32:58.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:58.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:58.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:32:58.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-18T06:32:58.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:32:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:32:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:32:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:58.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:58.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:58.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:32:58.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:32:58.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:58.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:32:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:32:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:32:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:32:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:32:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:32:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:32:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:32:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:32:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:32:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:32:59.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:59.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:59.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:32:59.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:32:59.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:32:59.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:32:59.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:32:59.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:32:59.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:32:59.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:32:59.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:33:01.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:33:01.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:01.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:01.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:01.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:33:01.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:01.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:33:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:01.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:01.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:01.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:01.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:01.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:01.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:01.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:01.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:01.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:01.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:01.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:01.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:01.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:01.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:33:01.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:01.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:33:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:33:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:33:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:33:05.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:33:05.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:05.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:05.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:05.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:05.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:33:05.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:05.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:33:05.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:33:05.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:05.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:33:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:33:05.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:33:06.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test 2026-02-18T06:33:06.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:06.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test' 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:33:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:33:06.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:06.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:06.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:33:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:06.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:06.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:06.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:06.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:06.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:06.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2026-02-18T06:33:06.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test 2026-02-18T06:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '' 2026-02-18T06:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:06.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:33:06.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:33:06.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:33:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:33:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:33:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:33:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=0, tag_tid=118, entry_tid=0' 2026-02-18T06:33:06.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test 2026-02-18T06:33:06.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test '..*' 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test.status 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test.status 2026-02-18T06:33:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:06.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:33:06.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2026-02-18T06:33:06.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:33:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:33:06.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2026-02-18T06:33:06.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.status 2026-02-18T06:33:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=0, tag_tid=118, entry_tid=0' 2026-02-18T06:33:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=0, tag_tid=118, entry_tid=0' -a 'object_number=0, tag_tid=118, entry_tid=0' = 'object_number=0, tag_tid=118, entry_tid=0' 2026-02-18T06:33:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-18T06:33:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:07.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-18T06:33:07.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:07.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:02 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:33:07.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:33:07.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2026-02-18T06:33:07.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:02' 2026-02-18T06:33:07.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:33:07.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:33:07.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:33:07.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:33:07.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:07.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:07.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:07.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:07.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:07.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:07.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:07.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:07.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:02' 2026-02-18T06:33:07.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-18 06:33:02' 2026-02-18T06:33:07.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:07.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:07.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:07.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:07.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:02' 2026-02-18T06:33:07.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:07.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:07.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:33:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:33:02cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2026-02-18 06:33:02' 2026-02-18T06:33:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:07.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:33:07.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:02' 2026-02-18T06:33:07.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:33:07.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:33:07.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:07.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:33:07.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:07.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:08.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:08.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:08.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: c5801b7e-e96a-45a4-9eeb-d2c6f33ee72a 2026-02-18T06:33:08.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:02 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:07' 2026-02-18T06:33:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:33:08.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:33:08.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:33:08.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2026-02-18T06:33:08.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:08.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:08.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:08.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:08.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:08.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:02' 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:33:02' 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:02' 2026-02-18T06:33:08.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:08.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:08.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:33:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}43444348mirror.0trial1302026-02-18 06:33:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:33:07' 2026-02-18T06:33:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:33:08.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:02' 2026-02-18T06:33:08.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:33:08.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:33:08.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:33:08.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:33:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: force promote' 2026-02-18T06:33:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:33:08.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:33:08.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:33:08 TEST: force promote 2026-02-18T06:33:08.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:33:08 TEST: force promote 2026-02-18T06:33:08.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ force_promote_image=test_force_promote 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test_force_promote journal 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:33:08.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test_force_promote 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:08.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_force_promote 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_force_promote 100 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:33:08.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:33:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:33:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:33:08.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:08.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:08.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-18T06:33:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:09.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:33:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:33:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:33:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:09.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:09.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:09.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:33:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:33:09.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:33:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:33:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:33:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:33:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:09.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:09.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:09.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:09.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:1 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:09.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:1 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:09.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:1 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:09.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:33:09.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:33:09.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:33:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:33:09.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:09.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:09.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:2 2026-02-18T06:33:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:2 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:2 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:09.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-18T06:33:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:33:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:33:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:09.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:33:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:33:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:33:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:33:09.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:09.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:09.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:3 2026-02-18T06:33:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:09.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:3 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:3 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-18T06:33:09.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:33:09.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:33:09.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:09.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:09.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:33:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:33:09.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:33:09.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:33:09.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:09.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:09.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:33:10.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:33:10.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:10.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:10.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:10.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:33:10.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:10.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:10.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:33:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:33:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:33:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:33:10.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:10.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:10.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:10.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:10.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:33:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:33:10.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:10.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:33:10.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:10.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote 2026-02-18T06:33:10.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:10.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:10.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote started 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_force_promote started 2026-02-18T06:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:33:11.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:11.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:11.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:11.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:11.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:11.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:11.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:11.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:11.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:11.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:11.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:11.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:11.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:11.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:11.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:11.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:11.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:11.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:11.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:11.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:11.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:11.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:11.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:11.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:11.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:11.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:11.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:11.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:11.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:11.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:11.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:11.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:11.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:11.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:11.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:11.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:12.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:12.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:12.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:33:13.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:13.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:13.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:13.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:13.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:13.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:13.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:13.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:13.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:13.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:13.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:13.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:13.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:13.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:13.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:13.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:13.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:13.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:13.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:13.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:13.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:13.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:13.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:13.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:13.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:13.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:13.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:13.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:13.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:13.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:13.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:13.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:13.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:13.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:13.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:13.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:13.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:13.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:13.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:13.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:13.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:13.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:13.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-18T06:33:13.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:13.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:13.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:13.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:33:15.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:15.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:15.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:15.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:15.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:15.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:15.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:15.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:15.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:15.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:15.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:33:15.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:33:15.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:15.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test_force_promote 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:33:15.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test_force_promote 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:33:15.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_force_promote 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_force_promote ']' 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_force_promote' 2026-02-18T06:33:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:33:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote 2026-02-18T06:33:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:33:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:15.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:15.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:15.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:15.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:15.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:15.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:15.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote 2026-02-18T06:33:15.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_force_promote 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_force_promote '' 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:15.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2026-02-18T06:33:15.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:15.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:15.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:15.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:15.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:15.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:33:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote 2026-02-18T06:33:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote 2026-02-18T06:33:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*' 2026-02-18T06:33:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:15.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:15.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote 2026-02-18T06:33:15.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:33:15.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:15.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2026-02-18T06:33:15.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:15.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:15.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:15.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote.status 2026-02-18T06:33:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_force_promote 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:16.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position '' 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2026-02-18T06:33:16.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote: 2026-02-18T06:33:16.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 3a68d97b-e191-49fa-bc2f-777c4bc7299e 2026-02-18T06:33:16.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:33:16.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:13 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:12' 2026-02-18T06:33:16.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:33:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:33:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:33:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:16.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:16.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:13' 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+replaying '2026-02-18 06:33:13' 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:13' 2026-02-18T06:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:33:16.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}43444348mirror.0trial1302026-02-18 06:33:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:33:12test_force_promote3a68d97b-e191-49fa-bc2f-777c4bc7299eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:33:13cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:33:12' 2026-02-18T06:33:16.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:16.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-18T06:33:16.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:13' 2026-02-18T06:33:16.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-18T06:33:16.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:33:16.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:16.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:33:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2026-02-18T06:33:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:16.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:16.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:17.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2026-02-18T06:33:17.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:17.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:17.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:17.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:17.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:17.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:17.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:17.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:17.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2026-02-18T06:33:17.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote: 2026-02-18T06:33:17.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 3a68d97b-e191-49fa-bc2f-777c4bc7299e 2026-02-18T06:33:17.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:17 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:13' 2026-02-18T06:33:17.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:33:17.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:33:17.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:33:17.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:33:17.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:17.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:17.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:17.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:17.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:17.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:17.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:17.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:17.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:17' 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2026-02-18 06:33:17' 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:17' 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:33:18.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK245714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:33:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2026-02-18 06:33:02test_force_promote3a68d97b-e191-49fa-bc2f-777c4bc7299eup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:33:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2026-02-18 06:33:13' 2026-02-18T06:33:18.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:18.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-18T06:33:18.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:17' 2026-02-18T06:33:18.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-18T06:33:18.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:33:18.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:18.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:33:18.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test_force_promote --force 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force=--force 2026-02-18T06:33:18.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote --force 2026-02-18T06:33:18.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:33:18.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote 2026-02-18T06:33:18.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:18.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote stopped 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:19.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_force_promote stopped 2026-02-18T06:33:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:33:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:19.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:19.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:19.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:33:19.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:33:19.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:19.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:33:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:20.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:33:20.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:20.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:20.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:33:20.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:20.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:20.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:33:20.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-18T06:33:20.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:20.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:33:20.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:33:20.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:20.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:33:20.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:20.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+stopped 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:21.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+stopped '' '' 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:21.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote: 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 3a68d97b-e191-49fa-bc2f-777c4bc7299e 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:18 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:17' 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:33:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:21.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:21.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:21.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:21.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:21.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:21.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:21.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:18' 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+stopped '2026-02-18 06:33:18' 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:18' 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:21.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:33:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK1143444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}43444348mirror.0trial1302026-02-18 06:33:02cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:33:17test_force_promote3a68d97b-e191-49fa-bc2f-777c4bc7299eup+stoppedstopped43444348mirror.0trial1302026-02-18 06:33:18cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:33:17' 2026-02-18T06:33:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-18T06:33:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:18' 2026-02-18T06:33:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-18T06:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:33:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2026-02-18T06:33:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2026-02-18T06:33:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:33:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2026-02-18T06:33:22.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote: 2026-02-18T06:33:22.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 3a68d97b-e191-49fa-bc2f-777c4bc7299e 2026-02-18T06:33:22.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:22.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:17 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:18' 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:33:22.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:33:22.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:22.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:22.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:22.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:22.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:22.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:22.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:22.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:17' 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2026-02-18 06:33:17' 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:17' 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:22.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:33:22.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK1145714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:33:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2026-02-18 06:33:02test_force_promote3a68d97b-e191-49fa-bc2f-777c4bc7299eup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:33:17cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedstopped2026-02-18 06:33:18' 2026-02-18T06:33:22.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:22.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-18T06:33:22.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:17' 2026-02-18T06:33:22.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-18T06:33:22.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:33:22.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:33:22.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:33:22.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:33:22.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:22.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:22.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster1 mirror test_force_promote 100 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:33:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:33:22.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:33:22.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:33:22.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:33:22.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_force_promote 100 2026-02-18T06:33:22.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:33:22.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:33:22.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:33:22.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:33:22.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:33:22.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:33:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:33:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:33:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster1 mirror test_force_promote 2026-02-18T06:33:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:33:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:22.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster1 mirror test_force_promote 2026-02-18T06:33:22.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:22.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_force_promote 2026-02-18T06:33:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 rm mirror/test_force_promote 2026-02-18T06:33:22.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:33:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test_force_promote 2026-02-18T06:33:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:33:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test_force_promote 2026-02-18T06:33:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote 2026-02-18T06:33:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_force_promote 2026-02-18T06:33:22.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test_force_promote 2026-02-18T06:33:22.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: cloned images' 2026-02-18T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:33:22 TEST: cloned images 2026-02-18T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:33:22 TEST: cloned images 2026-02-18T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - default' 2026-02-18T06:33:22.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:33:22.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:33:22.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:33:22 - default 2026-02-18T06:33:22.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:33:22 - default 2026-02-18T06:33:22.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ parent_image=test_parent 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ parent_snap=snap 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror_parent test_parent journal 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror_parent test_parent 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror_parent/test_parent 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror_parent 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ enable_mirror cluster2 mirror_parent test_parent journal 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:22.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:33:22.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent journal 2026-02-18T06:33:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Mirroring enabled 2026-02-18T06:33:22.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 info mirror_parent/test_parent 2026-02-18T06:33:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test_parent': 2026-02-18T06:33:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:33:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:33:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 0 2026-02-18T06:33:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 1635e9ffeccc 2026-02-18T06:33:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.1635e9ffeccc 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:33:22 2026 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:33:22 2026 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:33:22 2026 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 1635e9ffeccc 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: 92fea05d-9f97-41db-8337-238f6e280555 2026-02-18T06:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: true 2026-02-18T06:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror_parent test_parent 100 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror_parent/test_parent --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:33:22.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:33:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:33:22.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 10000.2 bytes/sec: 39 MiB/s 2026-02-18T06:33:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap 2026-02-18T06:33:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap 2026-02-18T06:33:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap 2026-02-18T06:33:23.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:33:23.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ protect_snapshot cluster2 mirror_parent test_parent snap 2026-02-18T06:33:23.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:23.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:23.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:23.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap 2026-02-18T06:33:23.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap protect mirror_parent/test_parent@snap 2026-02-18T06:33:23.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image=test_clone 2026-02-18T06:33:23.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone 2026-02-18T06:33:23.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:23.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:33:23.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:33:23.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap 2026-02-18T06:33:23.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:33:23.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone 2026-02-18T06:33:23.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:33:23.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone --image-feature layering,exclusive-lock,journaling 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_clone 100 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:33:23.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:33:23.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_clone --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:33:23.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:33:23.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:33:23.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 6250.12 bytes/sec: 24 MiB/s 2026-02-18T06:33:23.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:33:23.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ enable_mirror cluster2 mirror_parent test_parent journal 2026-02-18T06:33:23.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:23.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:23.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:23.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:33:23.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent journal 2026-02-18T06:33:23.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Mirroring enabled 2026-02-18T06:33:23.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 info mirror_parent/test_parent 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test_parent': 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 1 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 1635e9ffeccc 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.1635e9ffeccc 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:33:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:33:22 2026 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:33:22 2026 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:33:22 2026 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 1635e9ffeccc 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: 92fea05d-9f97-41db-8337-238f6e280555 2026-02-18T06:33:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: true 2026-02-18T06:33:23.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror_parent test_parent 2026-02-18T06:33:23.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:23.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror_parent test_parent started 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:23.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror_parent test_parent started 2026-02-18T06:33:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:33:24.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:24.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:24.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:24.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:24.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:24.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:24.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:24.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:24.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:24.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:24.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:24.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:24.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:24.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:24.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:24.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:24.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:24.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:24.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:24.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:24.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:24.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:33:25.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:25.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:25.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:25.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:25.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:25.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:25.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:25.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:25.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:25.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:25.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:25.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:25.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:25.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:25.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:25.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:25.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:25.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:25.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:25.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:25.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:25.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:33:27.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:27.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:27.971 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:27.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:27.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:27.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:28.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:28.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:28.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:28.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:28.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:28.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:28.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:28.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:28.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:28.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:28.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:28.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:28.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:28.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-18T06:33:28.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:28.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:28.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:28.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:33:32.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:32.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:32.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:32.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:32.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:32.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:32.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2026-02-18T06:33:32.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:32.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:33:32.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:33:32.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:32.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:33:32.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:33:32.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:33:32.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:32.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror_parent mirror_parent test_parent 2026-02-18T06:33:32.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror_parent 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror_parent 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror_parent mirror_parent test_parent 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror_parent 2026-02-18T06:33:32.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror_parent 2026-02-18T06:33:32.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:32.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:33:32.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:33:32.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:33:32.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:33:32.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror_parent test_parent 2026-02-18T06:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_parent ']' 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror_parent/test_parent' 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror_parent/test_parent 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror_parent/test_parent 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:32.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:32.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:32.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:32.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:32.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror_parent/test_parent 2026-02-18T06:33:32.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror_parent test_parent 2026-02-18T06:33:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror_parent 2026-02-18T06:33:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_parent 2026-02-18T06:33:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror_parent test_parent '' 2026-02-18T06:33:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror_parent 2026-02-18T06:33:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_parent 2026-02-18T06:33:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:33:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:32.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2026-02-18T06:33:32.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2026-02-18T06:33:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=52693b04-c110-4f0a-8ace-a7cde98ec3f8, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2026-02-18T06:33:32.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=4, entry_tid=3' 2026-02-18T06:33:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror_parent test_parent 2026-02-18T06:33:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror_parent 2026-02-18T06:33:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_parent 2026-02-18T06:33:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror_parent test_parent '..*' 2026-02-18T06:33:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror_parent 2026-02-18T06:33:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_parent 2026-02-18T06:33:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:33:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:32.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2026-02-18T06:33:32.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:32.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:32.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:32.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2026-02-18T06:33:32.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=52693b04-c110-4f0a-8ace-a7cde98ec3f8, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2026-02-18T06:33:32.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.status 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=4, entry_tid=3' 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=4, entry_tid=3' -a 'object_number=3, tag_tid=4, entry_tid=3' = 'object_number=3, tag_tid=4, entry_tid=3' 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror_parent test_parent 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position 2026-02-18T06:33:32.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:32.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:32.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:32.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:32.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:33:32.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:32.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:32.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position '' 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:33.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:33.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror_parent/test_parent 2026-02-18T06:33:33.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_parent: 2026-02-18T06:33:33.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 92fea05d-9f97-41db-8337-238f6e280555 2026-02-18T06:33:33.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:33:33.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2026-02-18T06:33:33.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:33.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:29 2026-02-18T06:33:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:33:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:32' 2026-02-18T06:33:33.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:33:33.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:33:33.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:33:33.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2026-02-18T06:33:33.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:33.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:33.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:33.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:33.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:33.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:33.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:33.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:29' 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror_parent test_parent up+replaying '2026-02-18 06:33:29' 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:29' 2026-02-18T06:33:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:33.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:33.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror_parent --verbose --format xml 2026-02-18T06:33:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143444360mirror.0trial13020.2.0-670-gadc19233trueOK43504377mirror.1trial13020.2.0-670-gadc19233falseOK43574374mirror.2trial13020.2.0-670-gadc19233falseOK43674378mirror.3trial13020.2.0-670-gadc19233falseOKtest_parent92fea05d-9f97-41db-8337-238f6e280555up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}43444360mirror.0trial1302026-02-18 06:33:29cluster25fb949e8-4977-49c4-a3d3-6ecb619df0ceup+stoppedlocal image is primary2026-02-18 06:33:32' 2026-02-18T06:33:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/last_update' 2026-02-18T06:33:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:29' 2026-02-18T06:33:33.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/state' 2026-02-18T06:33:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:33:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:33:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:33.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:33.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror_parent mirror_parent test_parent 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror_parent 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror_parent 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror_parent-test_parent.export 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror_parent-test_parent.export 2026-02-18T06:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:33.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.export 2026-02-18T06:33:33.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror_parent-test_parent.export 2026-02-18T06:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror_parent-test_parent.export 2026-02-18T06:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:33.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror_parent-test_parent.export 2026-02-18T06:33:33.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_parent-test_parent.export 2026-02-18T06:33:33.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror_parent/test_parent /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.export 2026-02-18T06:33:33.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:33:33.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror_parent/test_parent /tmp/tmp.mzUeR8FHMy/cluster1-mirror_parent-test_parent.export 2026-02-18T06:33:33.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:33:33.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_parent-test_parent.export 2026-02-18T06:33:33.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_parent-test_parent.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_parent-test_parent.export 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone started 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:33.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:33:33.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:33.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:33.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:34.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_clone started 2026-02-18T06:33:34.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:34.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:34.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:34.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:33:34.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:34.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:34.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone 2026-02-18T06:33:34.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:34.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:34.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:34.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:34.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2026-02-18T06:33:34.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:34.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:34.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:34.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:34.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:34.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:34.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:34.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2026-02-18T06:33:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2026-02-18T06:33:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:34.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:34.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:34.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:34.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:35.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:35.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:35.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:35.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone 2026-02-18T06:33:35.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2026-02-18T06:33:35.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:35.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone 2026-02-18T06:33:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:35.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:35.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:35.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:35.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:35.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:35.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:35.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone 2026-02-18T06:33:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2026-02-18T06:33:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:35.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:35.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:35.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:35.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:35.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:35.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:35.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone 2026-02-18T06:33:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2026-02-18T06:33:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:33:36.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:36.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:36.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:36.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:36.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:36.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:36.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:36.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:36.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:36.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:36.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:36.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:36.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:36.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:36.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2026-02-18T06:33:36.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:36.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:33:36.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:33:36.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:36.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:33:36.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:33:36.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test_clone 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test_clone 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:36.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:33:36.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:33:36.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:36.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:33:36.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:33:36.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:33:36.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:33:36.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:33:36.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_clone 2026-02-18T06:33:36.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_clone ']' 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_clone' 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_clone 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:33:36.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:33:36.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:36.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:36.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:36.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:33:36.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:36.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_clone 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:36.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:36.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:33:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:33:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:36.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:36.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:36.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:36.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:36.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_clone 2026-02-18T06:33:36.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:36.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_clone 2026-02-18T06:33:36.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:36.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:36.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone 2026-02-18T06:33:36.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_clone '' 2026-02-18T06:33:36.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone 2026-02-18T06:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_clone.status 2026-02-18T06:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_clone.status 2026-02-18T06:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:36.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.status 2026-02-18T06:33:36.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone 2026-02-18T06:33:36.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.status 2026-02-18T06:33:36.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:36.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:36.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:36.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:36.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:36.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.status 2026-02-18T06:33:36.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:33:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_clone 2026-02-18T06:33:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_clone '..*' 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_clone.status 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_clone.status 2026-02-18T06:33:36.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.status 2026-02-18T06:33:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone 2026-02-18T06:33:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.status 2026-02-18T06:33:36.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:33:36.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:33:36.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:33:36.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:36.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:33:36.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.status 2026-02-18T06:33:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:33:36.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_clone 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:36.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:33:36.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:36.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:36.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position '' 2026-02-18T06:33:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:33:37.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:33:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:37.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_clone 2026-02-18T06:33:37.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_clone: 2026-02-18T06:33:37.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 2e557f78-3ce2-4aac-85ea-b7cc3c101b9c 2026-02-18T06:33:37.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:33:37.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:33:37.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:33:37.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:35 2026-02-18T06:33:37.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:33:37.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:33:37.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:33:37.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:33:37.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:33:37' 2026-02-18T06:33:37.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:33:37.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:33:37.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:33:37.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:33:37.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:33:37.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:37.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:33:37.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:37.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:33:37.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:33:37.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:33:37.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:33:37.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:35' 2026-02-18T06:33:37.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_clone up+replaying '2026-02-18 06:33:35' 2026-02-18T06:33:37.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:33:35' 2026-02-18T06:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:33:37.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:33:37.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":2.4,"entries_behind_primary":0,"entries_per_second":0.1,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}43444348mirror.0trial1302026-02-18 06:33:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:33:37test_clone2e557f78-3ce2-4aac-85ea-b7cc3c101b9cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:33:35cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:33:37' 2026-02-18T06:33:37.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:33:37.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/last_update' 2026-02-18T06:33:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:33:35' 2026-02-18T06:33:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/state' 2026-02-18T06:33:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:33:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:33:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:33:37.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test_clone 2026-02-18T06:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test_clone.export 2026-02-18T06:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test_clone.export 2026-02-18T06:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.export 2026-02-18T06:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test_clone.export 2026-02-18T06:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test_clone.export 2026-02-18T06:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_clone.export 2026-02-18T06:33:37.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_clone.export 2026-02-18T06:33:37.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test_clone /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.export 2026-02-18T06:33:37.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:33:37.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test_clone /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_clone.export 2026-02-18T06:33:37.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:33:37.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_clone.export 2026-02-18T06:33:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_clone.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_clone.export 2026-02-18T06:33:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test_clone 2026-02-18T06:33:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:33:37.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test_clone 2026-02-18T06:33:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone 2026-02-18T06:33:37.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone 2026-02-18T06:33:37.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone 2026-02-18T06:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:33:37.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:37.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - clone v1' 2026-02-18T06:33:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:33:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:33:38 - clone v1 2026-02-18T06:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:33:38 - clone v1 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image_and_enable_mirror cluster1 mirror_parent test_parent snap mirror test_clone1 journal 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone1 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image cluster1 mirror_parent test_parent snap mirror test_clone1 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone1 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:33:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap mirror/test_clone1 --image-feature layering,exclusive-lock,journaling 2026-02-18T06:33:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:33:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:33:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:33:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:33:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image_and_enable_mirror cluster2 mirror_parent test_parent snap mirror test_clone_v1 journal --rbd-default-clone-format 1 2026-02-18T06:33:38.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone_v1 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone_v1 --rbd-default-clone-format 1 2026-02-18T06:33:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:33:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:33:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap 2026-02-18T06:33:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:33:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone_v1 2026-02-18T06:33:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:33:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone_v1 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 1 2026-02-18T06:33:38.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_clone_format cluster2 mirror test_clone_v1 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone_v1 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v1 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk 'BEGIN { 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 1 2026-02-18T06:33:38.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: $1 == "parent:" { 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: parent = $2 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: /op_features: .*clone-child/ { 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 2 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: END { 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: if (!parent) exit 1 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: print format 2026-02-18T06:33:38.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: }' 2026-02-18T06:33:38.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 1 2026-02-18T06:33:38.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v1 2026-02-18T06:33:38.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:38.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:38.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v1 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v1 started 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v1 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_clone_v1 started 2026-02-18T06:33:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v1 2026-02-18T06:33:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:33:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:39.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:39.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:39.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:39.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:39.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:39.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:39.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:39.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:39.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:39.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:39.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:39.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:39.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:39.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:39.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:39.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:39.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:39.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:39.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:39.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:39.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:39.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:39.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:39.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:39.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:39.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:39.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:39.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:39.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:39.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:39.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:39.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:39.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:39.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:39.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:39.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:39.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:39.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:39.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:39.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:39.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:40.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:40.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:40.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:40.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:40.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:40.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:40.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:40.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:40.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:40.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:40.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:40.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:40.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:40.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:40.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:40.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:40.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:40.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:40.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:40.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:40.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:40.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:40.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:40.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:40.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:40.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-18T06:33:40.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:40.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:40.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:40.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:33:42.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:42.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:42.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:42.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:42.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:42.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:42.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2026-02-18T06:33:42.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:33:42.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:33:42.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:33:42.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_clone_format cluster1 mirror test_clone_v1 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone_v1 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v1 2026-02-18T06:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk 'BEGIN { 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 1 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: $1 == "parent:" { 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: parent = $2 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: /op_features: .*clone-child/ { 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 2 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: END { 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: if (!parent) exit 1 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: print format 2026-02-18T06:33:42.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: }' 2026-02-18T06:33:42.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 1 2026-02-18T06:33:42.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test_clone_v1 2026-02-18T06:33:42.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:42.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:42.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v1 2026-02-18T06:33:42.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:33:42.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:42.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test_clone_v1 2026-02-18T06:33:42.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:42.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:42.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v1 2026-02-18T06:33:42.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v1 2026-02-18T06:33:42.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v1 2026-02-18T06:33:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:33:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster1 mirror test_clone1 2026-02-18T06:33:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone1 2026-02-18T06:33:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:33:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster1 mirror test_clone1 2026-02-18T06:33:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone1 2026-02-18T06:33:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone1 2026-02-18T06:33:42.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone1 2026-02-18T06:33:42.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:33:42.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ unprotect_snapshot_retry cluster2 mirror_parent test_parent snap 2026-02-18T06:33:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap 2026-02-18T06:33:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:33:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:33:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ unprotect_snapshot cluster2 mirror_parent test_parent snap 2026-02-18T06:33:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap 2026-02-18T06:33:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap unprotect mirror_parent/test_parent@snap 2026-02-18T06:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap 2026-02-18T06:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap 2026-02-18T06:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap 2026-02-18T06:33:43.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing snap: 100% complete...done. 2026-02-18T06:33:43.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - clone v2' 2026-02-18T06:33:43.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:33:43.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:33:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:33:43 - clone v2 2026-02-18T06:33:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:33:43 - clone v2 2026-02-18T06:33:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ parent_snap=snap_v2 2026-02-18T06:33:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:33:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:43.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:33:43.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:33:43.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_v2 2026-02-18T06:33:43.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2026-02-18T06:33:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:33:44.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:33:44.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image_and_enable_mirror cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 journal --rbd-default-clone-format 2 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap_v2 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone_v2 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:33:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap_v2 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone_v2 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:33:44.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_clone_format cluster2 mirror test_clone_v2 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone_v2 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v2 2026-02-18T06:33:44.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk 'BEGIN { 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 1 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: $1 == "parent:" { 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: parent = $2 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: /op_features: .*clone-child/ { 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 2 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: END { 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: if (!parent) exit 1 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: print format 2026-02-18T06:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: }' 2026-02-18T06:33:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 2 2026-02-18T06:33:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v2 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v2 started 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:44.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:33:45.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_clone_v2 started 2026-02-18T06:33:45.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:33:45.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:33:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:33:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:33:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:33:45.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:33:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:33:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:33:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:33:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:45.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:45.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:45.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:45.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:45.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:45.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:45.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:45.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:45.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:45.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:45.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:45.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:45.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:45.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:45.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:45.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:45.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:45.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:45.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:45.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:45.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:45.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:45.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:45.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:45.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:45.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:45.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:45.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:45.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:45.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:45.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:45.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:45.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:33:46.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:46.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:46.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:46.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:46.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:46.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:46.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:46.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:46.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:46.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:46.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:46.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:46.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:46.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:33:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:48.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:48.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:48.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:48.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:48.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:48.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:48.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:48.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:48.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:48.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:48.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:48.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:48.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:48.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:48.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:48.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:48.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:48.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:48.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:48.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:49.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:49.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:49.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:49.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:49.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:49.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:49.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:49.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:49.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:49.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:49.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:33:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:33:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:33:53.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:33:53.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:53.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:53.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:53.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:33:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:33:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:33:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:33:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:33:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:33:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:53.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:33:53.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:33:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:53.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:33:53.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:53.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:53.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:53.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:53.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:33:53.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:33:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:33:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:33:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:33:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:33:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:33:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:33:53.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:33:53.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:53.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:53.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:33:53.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:33:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-18T06:33:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:33:53.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:33:53.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:33:53.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 8 2026-02-18T06:34:01.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:01.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:34:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2026-02-18T06:34:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:34:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:34:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:01.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2026-02-18T06:34:01.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:34:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:34:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:34:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:34:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:34:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:34:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:34:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_clone_format cluster1 mirror test_clone_v2 2026-02-18T06:34:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone_v2 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v2 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk 'BEGIN { 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 1 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: $1 == "parent:" { 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: parent = $2 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: /op_features: .*clone-child/ { 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: format = 2 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: } 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: END { 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: if (!parent) exit 1 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: print format 2026-02-18T06:34:01.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: }' 2026-02-18T06:34:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 2 2026-02-18T06:34:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_v2 2026-02-18T06:34:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2026-02-18T06:34:01.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing snap: 100% complete...done. 2026-02-18T06:34:01.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:34:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:01.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 6 b429731e-03ab-4a57-a64d-e37647036b3d 128 MiB Wed Feb 18 06:33:44 2026 trash (user snap_v2) 2026-02-18T06:34:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:02.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:02.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:02.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:02.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:02.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:02.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:02.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:02.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:34:02.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:02.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:04.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:04.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:04.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:04.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:04.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:04.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:04.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:04.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 5 9c67683a-f97e-45c6-ab65-4cda851552f8 128 MiB Wed Feb 18 06:33:52 2026 trash (user snap_v2) 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test_clone_v2 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test_clone_v2 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:04.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:34:04.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v2 2026-02-18T06:34:04.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v2 2026-02-18T06:34:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test_clone_v2 deleted 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test_clone_v2 2026-02-18T06:34:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone_v2 2026-02-18T06:34:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2026-02-18T06:34:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:05.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2026-02-18T06:34:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:05.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:05.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:05.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test_clone_v2 deleted '' 2026-02-18T06:34:05.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:05.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:05.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test_clone_v2 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_clone_v2 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2026-02-18T06:34:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:05.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:05.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:05.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:05.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:05.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:05.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:05.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:05.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:05.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:05.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:05.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:06.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:06.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:06.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:06.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:06.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:06.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:06.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:06.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - clone v2 non-primary' 2026-02-18T06:34:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:34:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:34:06 - clone v2 non-primary 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:34:06 - clone v2 non-primary 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_v2 2026-02-18T06:34:06.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2026-02-18T06:34:07.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:07.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:34:07.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:07.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:07.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:07.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:07.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:07.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:07.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:07.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:08.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2026-02-18T06:34:08.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:34:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:34:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:08.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:10.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2026-02-18T06:34:10.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test_parent': 2026-02-18T06:34:10.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:34:10.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:34:10.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 1 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 110868bc1c5 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.110868bc1c5 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:33:24 2026 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:33:24 2026 2026-02-18T06:34:10.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:33:24 2026 2026-02-18T06:34:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:34:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 110868bc1c5 2026-02-18T06:34:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:34:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:34:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: 92fea05d-9f97-41db-8337-238f6e280555 2026-02-18T06:34:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image_and_enable_mirror cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 journal --rbd-default-clone-format 2 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap_v2 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:34:10.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone_v2 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ clone_image cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_pool=mirror_parent 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_image=test_parent 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local parent_snap=snap_v2 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_pool=mirror 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local clone_image=test_clone_v2 2026-02-18T06:34:10.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 6 2026-02-18T06:34:10.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_v2 2026-02-18T06:34:10.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2026-02-18T06:34:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing snap: 100% complete...done. 2026-02-18T06:34:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2026-02-18T06:34:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:11.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:11.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:11.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:34:11.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:11.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:11.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:11.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:12.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:12.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:12.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 6 5d61be83-edc1-4898-80f5-6085060d7823 128 MiB Wed Feb 18 06:34:09 2026 trash (user snap_v2) 2026-02-18T06:34:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster1 mirror test_clone_v2 2026-02-18T06:34:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:34:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:34:12.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:12.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster1 mirror test_clone_v2 2026-02-18T06:34:12.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:12.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:12.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_clone_v2 2026-02-18T06:34:12.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone_v2 2026-02-18T06:34:12.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone_v2 2026-02-18T06:34:13.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:34:13.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:13.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:13.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:13.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:13.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-18T06:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:14.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:14.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:14.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_v2 2026-02-18T06:34:14.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-18T06:34:14.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-18T06:34:14.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:14.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:14.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror_parent test_parent 2026-02-18T06:34:14.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:14.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:14.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:14.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:34:14.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:14.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror_parent test_parent 2026-02-18T06:34:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror_parent 2026-02-18T06:34:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_parent 2026-02-18T06:34:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror_parent/test_parent 2026-02-18T06:34:14.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror_parent/test_parent 2026-02-18T06:34:14.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:34:14.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:14.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: data pool' 2026-02-18T06:34:14.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:34:14.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:34:14 TEST: data pool 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:34:14 TEST: data pool 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ dp_image=test_data_pool 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test_data_pool journal 128 --data-pool mirror_parent 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:14.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test_data_pool 128 --data-pool mirror_parent 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:34:14.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 128 ']' 2026-02-18T06:34:14.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=128 2026-02-18T06:34:14.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:14.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/test_data_pool 2026-02-18T06:34:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:34:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:34:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:34:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:34:14.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_data_pool cluster2 mirror test_data_pool 2026-02-18T06:34:14.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:14.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:14.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:14.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 info mirror/test_data_pool 2026-02-18T06:34:14.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-18T06:34:14.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ data_pool=mirror_parent 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test mirror_parent = mirror_parent 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_data_pool 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_data_pool started 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:14.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_data_pool started 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:34:15.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:34:15.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:15.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:34:15.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:15.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:15.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:15.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:15.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:34:15.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:34:15.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:15.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:15.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:15.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:15.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:15.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:34:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:15.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:34:15.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:34:15.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:34:15.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:34:15.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:34:15.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:15.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:15.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:15.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:34:15.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:15.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:15.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:34:15.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:34:15.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:34:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:34:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:34:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:34:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:34:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:34:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:34:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:15.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:15.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:15.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:15.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:15.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:34:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:34:16.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:34:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:16.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:16.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:16.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:16.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:16.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:16.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:16.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:34:16.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:34:16.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:34:16.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:34:16.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:16.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:16.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:16.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:34:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:34:16.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:34:16.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:34:16.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:34:16.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:34:16.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:34:16.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:16.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:16.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:16.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:34:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:34:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:34:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:34:16.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:34:16.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:34:16.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:34:16.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:17.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-18T06:34:17.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:34:17.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:34:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:34:19.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:19.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:34:19.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:34:19.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:34:19.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:34:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:34:19.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:19.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2026-02-18T06:34:19.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:34:19.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:34:19.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:34:19.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:34:19.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:34:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:34:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:34:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_data_pool cluster1 mirror test_data_pool 2026-02-18T06:34:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 info mirror/test_data_pool 2026-02-18T06:34:19.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-18T06:34:19.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ data_pool=mirror_parent 2026-02-18T06:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test mirror_parent = mirror_parent 2026-02-18T06:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test_data_pool snap1 2026-02-18T06:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap1 2026-02-18T06:34:19.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_data_pool 100 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:19.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:19.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:19.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.45 bytes/sec: 30 MiB/s 2026-02-18T06:34:19.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test_data_pool snap2 2026-02-18T06:34:19.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:19.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:19.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:19.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:19.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap2 2026-02-18T06:34:20.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:20.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_data_pool 100 2026-02-18T06:34:20.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:20.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:20.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:20.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:20.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:20.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:20.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:20.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:20.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:20.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:20.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:34:20.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test_data_pool 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test_data_pool 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:34:20.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:34:20.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:20.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:34:20.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:34:20.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:34:20.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:34:20.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:34:20.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_data_pool 2026-02-18T06:34:20.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:20.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:20.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:20.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:34:20.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_data_pool ']' 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:20.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:34:20.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:20.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:20.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:20.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:20.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:20.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:34:20.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:34:20.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:20.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:20.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:20.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:20.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:20.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:20.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:20.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:20.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2026-02-18T06:34:20.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:20.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:20.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:20.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:34:20.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:20.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:20.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:20.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:20.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:20.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:20.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:34:20.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:34:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:20.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=5 2026-02-18T06:34:20.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:34:20.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:34:20.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:20.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:34:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:34:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:20.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:34:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:34:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:34:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=3 2026-02-18T06:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 5 = 1 -a 101 -le 3 2026-02-18T06:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:34:20.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:34:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_data_pool 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_data_pool ']' 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:34:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:34:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:34:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:34:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:34:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:34:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:20.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:20.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:20.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:20.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:20.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:20.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:34:20.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:20.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:20.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:20.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:21.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:21.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:21.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:34:21.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:21.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:21.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:21.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:21.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:21.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' '!=' 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:34:21.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:34:21.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:34:21.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:34:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=5 2026-02-18T06:34:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:34:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=3 2026-02-18T06:34:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:34:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:21.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:34:21.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:34:21.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:34:21.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=101 2026-02-18T06:34:21.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 5 = 3 -a 101 -le 101 2026-02-18T06:34:21.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:34:21.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:34:21.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:34:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_data_pool 2026-02-18T06:34:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:21.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_data_pool ']' 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:34:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:21.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:21.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:34:21.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:21.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:34:21.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:21.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:34:21.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:34:21.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:21.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:21.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:21.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:21.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2026-02-18T06:34:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:34:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:21.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:21.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:21.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:21.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:21.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:21.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:21.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2026-02-18T06:34:21.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:34:21.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:21.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:21.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:21.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:21.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:21.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' '!=' 'object_number=1, tag_tid=3, entry_tid=101' 2026-02-18T06:34:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:34:21.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:34:21.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_data_pool 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_data_pool ']' 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:21.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:34:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:34:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:34:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:34:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:34:21.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:34:21.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:34:21.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:34:21.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:34:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:21.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:34:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:34:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2026-02-18T06:34:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:34:21.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:34:21.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:34:21.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2026-02-18T06:34:21.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.status 2026-02-18T06:34:21.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:21.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101' 2026-02-18T06:34:21.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_data_pool 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:34:21.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:34:21.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:34:21.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:22.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position '' 2026-02-18T06:34:22.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:22.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:34:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:34:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:34:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:34:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:34:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_data_pool 2026-02-18T06:34:22.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_data_pool: 2026-02-18T06:34:22.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 2a93d24a-4218-4315-b04f-3c5198be7ef5 2026-02-18T06:34:22.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":138181.33,"entries_behind_primary":0,"entries_per_second":36.33,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:34:22 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:34:22' 2026-02-18T06:34:22.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:34:22.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:34:22.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:34:22.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":138181.33,"entries_behind_primary":0,"entries_per_second":36.33,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2026-02-18T06:34:22.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:34:22.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:34:22.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:34:22.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:34:22.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:34:22.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:34:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:34:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:34:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:34:22' 2026-02-18T06:34:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_data_pool up+replaying '2026-02-18 06:34:22' 2026-02-18T06:34:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:22.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:34:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:34:22' 2026-02-18T06:34:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:34:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:34:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:34:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}43444348mirror.0trial1302026-02-18 06:34:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:34:22test_data_pool2a93d24a-4218-4315-b04f-3c5198be7ef5up+replayingreplaying, {"bytes_per_second":138181.33,"entries_behind_primary":0,"entries_per_second":36.33,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}43444348mirror.0trial1302026-02-18 06:34:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:34:22' 2026-02-18T06:34:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:34:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2026-02-18T06:34:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:34:22' 2026-02-18T06:34:23.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2026-02-18T06:34:23.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:34:23.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:34:23.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:34:23.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:34:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test_data_pool@snap1 2026-02-18T06:34:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:34:23.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:34:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:34:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:34:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool@snap1 2026-02-18T06:34:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap1 /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:34:23.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap1 /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:34:23.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap1.export 2026-02-18T06:34:23.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:23.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test_data_pool@snap2 2026-02-18T06:34:23.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:34:23.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:34:23.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:23.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:34:23.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:34:23.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool@snap2 2026-02-18T06:34:23.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:23.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:23.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap2 /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:34:23.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap2 /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:34:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool@snap2.export 2026-02-18T06:34:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test_data_pool 2026-02-18T06:34:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:34:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:34:23.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:34:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:34:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test_data_pool.export 2026-02-18T06:34:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test_data_pool.export 2026-02-18T06:34:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.export 2026-02-18T06:34:23.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test_data_pool.export 2026-02-18T06:34:23.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test_data_pool.export 2026-02-18T06:34:23.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:34:23.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool.export 2026-02-18T06:34:23.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool.export 2026-02-18T06:34:23.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.export 2026-02-18T06:34:23.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:34:23.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool.export 2026-02-18T06:34:23.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:34:23.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool.export 2026-02-18T06:34:23.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_data_pool.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_data_pool.export 2026-02-18T06:34:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test_data_pool 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test_data_pool 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:23.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:34:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_data_pool 2026-02-18T06:34:25.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-18T06:34:25.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test_data_pool 2026-02-18T06:34:25.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:34:25.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:25.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: disable mirroring / delete non-primary image' 2026-02-18T06:34:25.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:34:25.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:34:25.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:34:25 TEST: disable mirroring / delete non-primary image 2026-02-18T06:34:25.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:34:25 TEST: disable mirroring / delete non-primary image 2026-02-18T06:34:25.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image2=test2 2026-02-18T06:34:25.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image3=test3 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image4=test4 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image5=test5 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test2 journal 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:25.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test2 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:34:25.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test2 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test2 100 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:25.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:25.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:25.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.45 bytes/sec: 30 MiB/s 2026-02-18T06:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test2 snap1 2026-02-18T06:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:25.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:25.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:25.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:25.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap1 2026-02-18T06:34:26.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:26.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test2 snap2 2026-02-18T06:34:26.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:26.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:26.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:26.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:26.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap2 2026-02-18T06:34:27.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' test2 = test4 ']' 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' test2 = test5 ']' 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test2 100 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:27.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:27.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:27.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:27.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:27.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test2 present 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:27.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:27.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:27.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:27.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:27.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:27.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:27.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id=10fc850f85a9 2026-02-18T06:34:27.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:27.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:27.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 present 10fc850f85a9 2026-02-18T06:34:27.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:27.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc850f85a9 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:27.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc850f85a9 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc850f85a9 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 10fc850f85a9 -o 10fc850f85a9 = 10fc850f85a9 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:27.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap2 2026-02-18T06:34:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:28.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2026-02-18T06:34:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:34:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:34:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:30.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2026-02-18T06:34:30.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test2': 2026-02-18T06:34:30.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:34:30.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 2 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 10fc850f85a9 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.10fc850f85a9 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:34:27 2026 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:34:27 2026 2026-02-18T06:34:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:34:27 2026 2026-02-18T06:34:30.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:34:30.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 10fc850f85a9 2026-02-18T06:34:30.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:34:30.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:34:30.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: 45bc9c10-7caf-4608-a9e6-8bacf9d53cab 2026-02-18T06:34:30.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test3 journal 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:30.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test3 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:34:30.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test3 2026-02-18T06:34:30.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:34:30.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:30.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test3 100 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 10000.2 bytes/sec: 39 MiB/s 2026-02-18T06:34:30.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test3 snap1 2026-02-18T06:34:30.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:30.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:30.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:30.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:30.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap1 2026-02-18T06:34:30.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:30.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test3 snap2 2026-02-18T06:34:30.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap2 2026-02-18T06:34:31.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' test3 = test4 ']' 2026-02-18T06:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' test3 = test5 ']' 2026-02-18T06:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test3 100 2026-02-18T06:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:31.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:31.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 5882.46 bytes/sec: 23 MiB/s 2026-02-18T06:34:31.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test3 present 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test3 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:31.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test3 2026-02-18T06:34:31.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-18T06:34:31.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:31.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test3: (2) No such file or directory 2026-02-18T06:34:31.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:31.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:31.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:31.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test3 present '' 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test3 2026-02-18T06:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test3 2026-02-18T06:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-18T06:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:31.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test3: (2) No such file or directory 2026-02-18T06:34:31.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:31.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:31.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:31.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:31.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test3 present '' 2026-02-18T06:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test3 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:32.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test3 2026-02-18T06:34:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-18T06:34:32.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:32.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test3: (2) No such file or directory 2026-02-18T06:34:32.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:32.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:32.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:32.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:32.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test3 present '' 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:34.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test3 2026-02-18T06:34:34.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:34.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:34.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test3 2026-02-18T06:34:34.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-18T06:34:34.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:34.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc365aa15 2026-02-18T06:34:34.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc365aa15 2026-02-18T06:34:34.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 10fc365aa15 2026-02-18T06:34:34.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:34:34.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test3 snap2 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap2 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:34.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:35.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2 2026-02-18T06:34:35.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test3': 2026-02-18T06:34:35.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 3 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 10fc365aa15 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.10fc365aa15 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling, operations 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: snap-trash 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:34:32 2026 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:34:32 2026 2026-02-18T06:34:35.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:34:32 2026 2026-02-18T06:34:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:34:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 10fc365aa15 2026-02-18T06:34:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:34:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:34:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: 5230fa2c-5ad7-4f2e-b5c5-fde859442efb 2026-02-18T06:34:35.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test4 journal 2026-02-18T06:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test4 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:34:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:34:35.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test4 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test4 100 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:35.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:35.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:34:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test4 snap1 2026-02-18T06:34:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:35.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:35.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:35.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap1 2026-02-18T06:34:36.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:36.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test4 snap2 2026-02-18T06:34:36.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:36.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:36.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:36.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:36.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap2 2026-02-18T06:34:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' test4 = test4 ']' 2026-02-18T06:34:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ protect_snapshot cluster2 mirror test4 snap1 2026-02-18T06:34:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:36.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:36.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:36.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:36.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap1 2026-02-18T06:34:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ protect_snapshot cluster2 mirror test4 snap2 2026-02-18T06:34:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:36.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:36.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap2 2026-02-18T06:34:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test4 100 2026-02-18T06:34:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:36.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:36.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:36.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test4 present 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:36.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:34:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:34:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:34:36.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-18T06:34:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test4 present '' 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:34:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:36.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-18T06:34:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:36.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test4 present '' 2026-02-18T06:34:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:37.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:34:37.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:37.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fcff219fe7 2026-02-18T06:34:37.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fcff219fe7 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 10fcff219fe7 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap2 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:37.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:37.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:38.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2026-02-18T06:34:38.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:34:38.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:34:38.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:38.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2026-02-18T06:34:40.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test4': 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 2 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 10fcff219fe7 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.10fcff219fe7 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:34:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:34:37 2026 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:34:37 2026 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:34:37 2026 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 10fcff219fe7 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: 2b53b1fa-3770-4180-8fba-7756de5beb70 2026-02-18T06:34:40.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:34:40.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test5 journal 2026-02-18T06:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test5 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:34:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:34:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:34:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test5 2026-02-18T06:34:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:34:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:34:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:34:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:34:40.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test5 100 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:40.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:34:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test5 snap1 2026-02-18T06:34:40.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:40.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap1 2026-02-18T06:34:41.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test5 snap2 2026-02-18T06:34:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap2 2026-02-18T06:34:41.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:34:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' test5 = test4 ']' 2026-02-18T06:34:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' test5 = test5 ']' 2026-02-18T06:34:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ protect_snapshot cluster2 mirror test5 snap1 2026-02-18T06:34:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap1 2026-02-18T06:34:41.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ protect_snapshot cluster2 mirror test5 snap2 2026-02-18T06:34:41.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:41.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:41.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:41.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:41.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap2 2026-02-18T06:34:41.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test5 100 2026-02-18T06:34:41.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:41.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:41.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:41.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:34:41.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:34:41.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:41.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:34:41.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:34:41.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:34:41.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:34:41.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 5555.66 bytes/sec: 22 MiB/s 2026-02-18T06:34:41.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:34:41.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test5 present 2026-02-18T06:34:41.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:41.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:41.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:41.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test5 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test5 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-18T06:34:41.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:41.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test5: (2) No such file or directory 2026-02-18T06:34:41.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:41.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:41.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test5 present '' 2026-02-18T06:34:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test5 2026-02-18T06:34:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test5 2026-02-18T06:34:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-18T06:34:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:41.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test5: (2) No such file or directory 2026-02-18T06:34:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:42.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test5 present '' 2026-02-18T06:34:42.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:42.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:42.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:42.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:42.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:42.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test5 2026-02-18T06:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test5 2026-02-18T06:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-18T06:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:42.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test5: (2) No such file or directory 2026-02-18T06:34:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:42.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test5 present '' 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:44.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test5 2026-02-18T06:34:44.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:44.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:44.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test5 2026-02-18T06:34:44.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-18T06:34:44.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc49fce85e 2026-02-18T06:34:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc49fce85e 2026-02-18T06:34:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 10fc49fce85e 2026-02-18T06:34:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:34:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:34:44.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test5 snap2 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap2 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:34:44.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:45.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2 2026-02-18T06:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test5': 2026-02-18T06:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 2 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 10fc49fce85e 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.10fc49fce85e 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:34:42 2026 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:34:42 2026 2026-02-18T06:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:34:42 2026 2026-02-18T06:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 10fc49fce85e 2026-02-18T06:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: cd6a440b-688e-47c6-bed9-8033787fbb0e 2026-02-18T06:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:34:45.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:45.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_pool_mirror_mode cluster2 mirror image 2026-02-18T06:34:45.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:45.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:45.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=image 2026-02-18T06:34:45.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2026-02-18T06:34:45.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:note: changing mirroring mode from pool to image 2026-02-18T06:34:45.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image4} 2026-02-18T06:34:45.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ disable_mirror cluster2 mirror test2 2026-02-18T06:34:45.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:45.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:45.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test2 2026-02-18T06:34:45.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Mirroring disabled 2026-02-18T06:34:45.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image4} 2026-02-18T06:34:45.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ disable_mirror cluster2 mirror test4 2026-02-18T06:34:45.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:45.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:45.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:45.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test4 2026-02-18T06:34:46.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Mirroring disabled 2026-02-18T06:34:46.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ unprotect_snapshot cluster2 mirror test5 snap1 2026-02-18T06:34:46.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:46.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:46.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap1 2026-02-18T06:34:46.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ unprotect_snapshot cluster2 mirror test5 snap2 2026-02-18T06:34:46.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:46.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap2 2026-02-18T06:34:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image3} ${image5} 2026-02-18T06:34:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_snapshot cluster2 mirror test3 snap1 2026-02-18T06:34:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:46.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap1 2026-02-18T06:34:46.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing snap: 100% complete...done. 2026-02-18T06:34:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_snapshot cluster2 mirror test3 snap2 2026-02-18T06:34:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap2 2026-02-18T06:34:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing snap: 100% complete...done. 2026-02-18T06:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test3 2026-02-18T06:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test3 2026-02-18T06:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test3 2026-02-18T06:34:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test3 2026-02-18T06:34:47.617 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:34:47.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:47.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image3} ${image5} 2026-02-18T06:34:47.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_snapshot cluster2 mirror test5 snap1 2026-02-18T06:34:47.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:34:47.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap1 2026-02-18T06:34:48.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing snap: 100% complete...done. 2026-02-18T06:34:48.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_snapshot cluster2 mirror test5 snap2 2026-02-18T06:34:48.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:48.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:48.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:48.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:34:48.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap2 2026-02-18T06:34:49.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing snap: 100% complete...done. 2026-02-18T06:34:49.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test5 2026-02-18T06:34:49.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:49.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:49.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:49.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:34:49.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:34:49.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test5 2026-02-18T06:34:49.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:49.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:49.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:49.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test5 2026-02-18T06:34:49.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test5 2026-02-18T06:34:49.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test2 deleted 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:49.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:49.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:49.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:49.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id=10fc850f85a9 2026-02-18T06:34:49.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:49.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 deleted 10fc850f85a9 2026-02-18T06:34:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:49.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc850f85a9 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc850f85a9 2026-02-18T06:34:49.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc850f85a9 2026-02-18T06:34:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 10fc850f85a9 -o 10fc850f85a9 = 10fc850f85a9 2026-02-18T06:34:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:34:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:34:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:50.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 deleted 10fc850f85a9 2026-02-18T06:34:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:34:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc850f85a9 2026-02-18T06:34:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:50.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:50.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:50.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:50.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:50.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:50.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:50.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc850f85a9 2026-02-18T06:34:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc850f85a9 2026-02-18T06:34:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 10fc850f85a9 -o 10fc850f85a9 = 10fc850f85a9 2026-02-18T06:34:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:34:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:34:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:52.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 deleted 10fc850f85a9 2026-02-18T06:34:52.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc850f85a9 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:52.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test2: (2) No such file or directory 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test3 deleted 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:52.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test3 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test3 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-18T06:34:52.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test3: (2) No such file or directory 2026-02-18T06:34:52.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:52.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:52.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:52.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test3 deleted '' 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test3 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test3 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test3 2026-02-18T06:34:52.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-18T06:34:52.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:53.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test3: (2) No such file or directory 2026-02-18T06:34:53.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test4 deleted 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:53.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:34:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:34:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:34:53.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:53.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-18T06:34:53.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test4 deleted '' 2026-02-18T06:34:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:34:53.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test5 deleted 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test5 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test5 2026-02-18T06:34:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-18T06:34:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:53.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test5: (2) No such file or directory 2026-02-18T06:34:53.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:53.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:53.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:53.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test5 deleted '' 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test5 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test5 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test5 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-18T06:34:53.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:53.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test5: (2) No such file or directory 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:53.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=pool 2026-02-18T06:34:53.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2026-02-18T06:34:53.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:note: changing mirroring mode from image to pool 2026-02-18T06:34:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image4} 2026-02-18T06:34:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ enable_journaling cluster2 mirror test2 2026-02-18T06:34:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:34:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:53.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 feature enable mirror/test2 journaling 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test2 present 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:53.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test2: (2) No such file or directory 2026-02-18T06:34:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:34:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:34:53.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 present '' 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:53.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test2: (2) No such file or directory 2026-02-18T06:34:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:53.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:34:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 present '' 2026-02-18T06:34:54.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test2: (2) No such file or directory 2026-02-18T06:34:54.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:34:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 present '' 2026-02-18T06:34:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:34:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:34:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:34:56.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:34:56.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:34:56.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test2: (2) No such file or directory 2026-02-18T06:34:56.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:34:56.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:34:56.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:34:56.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:34:56.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:35:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test2 present '' 2026-02-18T06:35:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:00.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test2 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:00.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-18T06:35:00.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:35:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fceba6175e 2026-02-18T06:35:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fceba6175e 2026-02-18T06:35:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 10fceba6175e 2026-02-18T06:35:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:35:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:35:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2026-02-18T06:35:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap2 2026-02-18T06:35:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:35:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:35:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:01.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2026-02-18T06:35:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:35:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:35:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:35:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:35:03.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2026-02-18T06:35:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test2': 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 2 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 10fceba6175e 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.10fceba6175e 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:35:00 2026 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:35:00 2026 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:35:00 2026 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 10fceba6175e 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: b3718e88-35d0-42ca-8167-370fab7f7682 2026-02-18T06:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test2 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test2 started 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:35:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:35:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:04.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test2 started 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test2 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:35:04.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:35:04.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:04.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:35:04.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:35:04.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:04.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:35:04.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:04.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:35:04.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:04.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test2 2026-02-18T06:35:04.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:35:04.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:35:04.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:35:04.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:35:04.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:04.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:04.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:04.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:04.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test2 2026-02-18T06:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:35:04.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:35:04.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:35:04.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:35:04.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:04.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test2 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test2 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:35:04.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test2 2026-02-18T06:35:04.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:04.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:04.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:04.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test2 ']' 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test2' 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:04.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:04.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:04.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:04.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:04.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:04.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2 2026-02-18T06:35:05.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test2 2026-02-18T06:35:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test2 '' 2026-02-18T06:35:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:35:05.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test2.status 2026-02-18T06:35:05.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test2.status 2026-02-18T06:35:05.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:05.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2026-02-18T06:35:05.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:05.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:05.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:05.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:05.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test2 2026-02-18T06:35:05.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test2 '..*' 2026-02-18T06:35:05.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:35:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test2.status 2026-02-18T06:35:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test2.status 2026-02-18T06:35:05.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:05.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2026-02-18T06:35:05.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:05.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:05.072 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:05.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' '' 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = '' 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:35:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:35:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=1 2026-02-18T06:35:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:35:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:35:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:35:05.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:35:05.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:05.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:35:05.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:35:05.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=3 2026-02-18T06:35:05.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2026-02-18T06:35:05.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:35:05.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:35:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=1 2026-02-18T06:35:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 1 -a 3 -le 1 2026-02-18T06:35:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:35:05.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:35:05.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:35:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test2 2026-02-18T06:35:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test2 ']' 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test2' 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:05.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2 2026-02-18T06:35:05.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test2 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test2 '' 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:35:05.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test2.status 2026-02-18T06:35:05.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test2.status 2026-02-18T06:35:05.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:05.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2026-02-18T06:35:05.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:05.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:05.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:05.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:05.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test2 2026-02-18T06:35:05.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test2 '..*' 2026-02-18T06:35:05.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:05.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test2 2026-02-18T06:35:05.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:35:05.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test2.status 2026-02-18T06:35:05.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test2.status 2026-02-18T06:35:05.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:05.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2026-02-18T06:35:05.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:05.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:05.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:05.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:05.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.status 2026-02-18T06:35:05.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test2 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test2.export 2026-02-18T06:35:05.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test2.export 2026-02-18T06:35:05.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:05.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.export 2026-02-18T06:35:05.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test2.export 2026-02-18T06:35:05.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test2.export 2026-02-18T06:35:05.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:05.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test2.export 2026-02-18T06:35:05.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test2.export 2026-02-18T06:35:05.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test2 /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.export 2026-02-18T06:35:05.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:35:05.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test2 /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test2.export 2026-02-18T06:35:05.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:35:05.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test2.export 2026-02-18T06:35:05.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test2.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test2.export 2026-02-18T06:35:05.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:05.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image4} 2026-02-18T06:35:05.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ enable_journaling cluster2 mirror test4 2026-02-18T06:35:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 feature enable mirror/test4 journaling 2026-02-18T06:35:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test4 present 2026-02-18T06:35:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:35:05.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:35:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:35:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-18T06:35:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:35:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:35:05.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:35:05.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test4 present '' 2026-02-18T06:35:05.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:05.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:35:05.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:35:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-18T06:35:05.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:35:05.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:35:05.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:35:05.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:35:05.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:06.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test4 present '' 2026-02-18T06:35:06.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:06.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:35:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:35:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-18T06:35:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:35:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:35:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:35:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:35:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test4 present '' 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:35:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:35:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test4 2026-02-18T06:35:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-18T06:35:08.897 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:35:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fce2e13bc9 2026-02-18T06:35:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fce2e13bc9 2026-02-18T06:35:08.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 10fce2e13bc9 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap2 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:35:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2026-02-18T06:35:09.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:35:09.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:35:09.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:35:09.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:35:11.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2026-02-18T06:35:11.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test4': 2026-02-18T06:35:11.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:35:11.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:35:11.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 3 2026-02-18T06:35:11.988 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 10fce2e13bc9 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.10fce2e13bc9 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling, operations 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: snap-trash 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:35:07 2026 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:35:07 2026 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:35:07 2026 2026-02-18T06:35:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: True 2026-02-18T06:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 10fce2e13bc9 2026-02-18T06:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: 6043cca4-bf88-4a33-92cb-fd23ce062877 2026-02-18T06:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test4 2026-02-18T06:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test4 started 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test4 started 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:35:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test4 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:35:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:35:12.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test4 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:12.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:35:12.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:35:12.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:12.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:12.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:35:12.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:35:12.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:35:12.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:35:12.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:13.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:13.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test4 2026-02-18T06:35:13.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:35:13.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:35:13.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:35:13.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:35:13.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:35:13.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:35:13.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:35:13.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:13.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test4 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test4 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:13.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:13.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:35:13.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:35:13.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:35:13.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:35:13.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:35:13.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test4 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test4 ']' 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test4' 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test4 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:35:13.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:13.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:13.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:13.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test4 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:13.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:13.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:13.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:13.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:13.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:13.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test4 2026-02-18T06:35:13.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test4 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test4 '' 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:13.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:35:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test4.status 2026-02-18T06:35:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test4.status 2026-02-18T06:35:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:13.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.status 2026-02-18T06:35:13.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2026-02-18T06:35:13.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.status 2026-02-18T06:35:13.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:13.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:13.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:13.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:13.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:13.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.status 2026-02-18T06:35:13.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test4 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test4 '..*' 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test4 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test4.status 2026-02-18T06:35:13.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test4.status 2026-02-18T06:35:13.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:13.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.status 2026-02-18T06:35:13.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.status 2026-02-18T06:35:13.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2026-02-18T06:35:13.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:13.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:13.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:13.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:13.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:35:13.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.status 2026-02-18T06:35:13.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:13.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:35:13.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:13.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test4 2026-02-18T06:35:13.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test4.export 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test4.export 2026-02-18T06:35:13.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:13.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.export 2026-02-18T06:35:13.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test4.export 2026-02-18T06:35:13.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test4.export 2026-02-18T06:35:13.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:13.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test4.export 2026-02-18T06:35:13.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test4.export 2026-02-18T06:35:13.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test4 /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.export 2026-02-18T06:35:13.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:35:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test4 /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test4.export 2026-02-18T06:35:13.578 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:35:13.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test4.export 2026-02-18T06:35:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test4.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test4.export 2026-02-18T06:35:13.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:13.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: remove mirroring pool' 2026-02-18T06:35:13.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:35:13.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:35:13.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:35:13 TEST: remove mirroring pool 2026-02-18T06:35:13.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:35:13 TEST: remove mirroring pool 2026-02-18T06:35:13.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pool=pool_to_remove 2026-02-18T06:35:13.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2026-02-18T06:35:13.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:35:13.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster1 osd pool create pool_to_remove 16 16 2026-02-18T06:35:14.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'pool_to_remove' already exists 2026-02-18T06:35:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:35:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 pool init pool_to_remove 2026-02-18T06:35:17.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable pool_to_remove pool 2026-02-18T06:35:17.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2026-02-18T06:35:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:35:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster2 osd pool create pool_to_remove 16 16 2026-02-18T06:35:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'pool_to_remove' already exists 2026-02-18T06:35:18.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:35:18.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 pool init pool_to_remove 2026-02-18T06:35:21.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable pool_to_remove pool 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_add cluster1 pool_to_remove cluster2 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local client_cluster=cluster2 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster2 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid_var_name 2026-02-18T06:35:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:35:21.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local error_code 2026-02-18T06:35:21.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local peer_uuid 2026-02-18T06:35:21.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 16 32 2026-02-18T06:35:21.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:35:21.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool peer add pool_to_remove cluster2 2026-02-18T06:35:21.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_uuid=2c3bb120-c8df-4f5a-a73d-28354006fafd 2026-02-18T06:35:21.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:35:21.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:35:21.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 17 ']' 2026-02-18T06:35:21.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 -eq 0 2026-02-18T06:35:21.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_add cluster2 pool_to_remove cluster1 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local client_cluster=cluster1 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_cluster=cluster1 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local uuid_var_name 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:35:21.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local error_code 2026-02-18T06:35:21.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local peer_uuid 2026-02-18T06:35:21.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 16 32 2026-02-18T06:35:21.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:35:21.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool peer add pool_to_remove cluster1 2026-02-18T06:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ peer_uuid=598542b4-3dc5-4944-8dc1-f1b7452f9ad6 2026-02-18T06:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 17 ']' 2026-02-18T06:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 -eq 0 2026-02-18T06:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rdp_image=test_remove_data_pool 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 pool_to_remove test 128 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 128 ']' 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=128 2026-02-18T06:35:21.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling pool_to_remove/test 2026-02-18T06:35:21.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test_remove_data_pool 128 --data-pool pool_to_remove 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 128 ']' 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=128 2026-02-18T06:35:21.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool pool_to_remove mirror/test_remove_data_pool 2026-02-18T06:35:21.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 pool_to_remove test 100 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:35:21.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench pool_to_remove/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:35:21.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:35:21.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:35:21.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:35:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_remove_data_pool 100 2026-02-18T06:35:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:35:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:35:21.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:35:21.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:35:21.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_remove_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:35:21.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:35:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:35:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:35:21.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 pool_to_remove pool_to_remove test 2026-02-18T06:35:21.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:21.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:21.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=pool_to_remove 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=pool_to_remove 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 pool_to_remove pool_to_remove test 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=pool_to_remove 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=pool_to_remove 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:35:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:35:21.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:35:21.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:35:21.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 pool_to_remove test 2026-02-18T06:35:21.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:21.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush pool_to_remove/test' 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:21.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:21.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:21.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:21.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:21.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:21.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:21.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:21.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:21.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:21.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:21.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:21.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:21.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:35:21.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:35:21.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:21.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:21.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:21.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:21.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:21.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:21.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:21.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:21.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:21.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:21.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:21.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:35:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:21.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:21.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:21.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:35:21.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:21.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:21.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:21.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:21.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:21.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:21.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:21.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:21.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:22.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:22.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:23.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:23.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:23.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:23.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:23.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:23.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:23.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:35:23.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:23.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:23.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:23.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:23.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:35:23.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:23.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:23.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:23.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:23.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:23.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:23.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:23.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:35:23.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:23.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:23.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:23.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:23.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:23.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:23.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:23.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:35:23.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:23.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:23.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:23.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:23.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:23.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:35:23.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:23.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:23.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:23.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:23.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:23.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:23.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:23.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:23.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:35:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:25.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:25.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:25.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:25.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:25.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:25.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:25.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:25.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:25.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:35:25.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:25.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:25.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:35:25.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:25.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:25.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:35:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:25.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:25.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:25.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:35:25.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:25.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:25.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:25.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:25.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:25.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:25.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:25.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:25.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:25.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:25.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:25.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:35:25.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:25.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:25.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:25.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:25.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:25.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:25.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:25.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:25.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:35:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:29.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:29.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:29.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:35:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:35:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:29.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:29.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:35:29.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:35:29.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:29.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:29.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:29.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:29.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:35:29.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:29.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:29.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:35:29.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:29.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror flush : flush rbd mirror 2026-02-18T06:35:29.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:35:29.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:35:29.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:35:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2026-02-18T06:35:29.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 pool_to_remove test 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=pool_to_remove 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 pool_to_remove test '' 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=pool_to_remove 2026-02-18T06:35:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:35:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:35:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2026-02-18T06:35:29.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:29.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:29.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:29.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:29.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=141b3c3b-ec90-4dfb-9025-70649cbdca91, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:29.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:35:29.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 pool_to_remove test 2026-02-18T06:35:29.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:29.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=pool_to_remove 2026-02-18T06:35:29.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:35:29.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 pool_to_remove test '..*' 2026-02-18T06:35:29.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:29.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=pool_to_remove 2026-02-18T06:35:29.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:35:29.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:35:29.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:29.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2026-02-18T06:35:29.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:29.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=141b3c3b-ec90-4dfb-9025-70649cbdca91, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:29.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-pool_to_remove-test.status 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 pool_to_remove test 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position 2026-02-18T06:35:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:30.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position '' 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:35:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status pool_to_remove/test 2026-02-18T06:35:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:35:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 2034a92f-64c1-454c-ab67-4a6055d08190 2026-02-18T06:35:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:35:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:35:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.3 on trial130 2026-02-18T06:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:35:27 2026-02-18T06:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:35:27' 2026-02-18T06:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:35:30.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:35:30.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:35:30.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:35:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:35:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:35:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:35:30.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:35:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:35:30.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.3 on trial130 2026-02-18T06:35:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:35:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:35:30.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:35:27' 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 pool_to_remove test up+replaying '2026-02-18 06:35:27' 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=pool_to_remove 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:35:27' 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:35:30.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status pool_to_remove --verbose --format xml 2026-02-18T06:35:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK143445635mirror.0trial13020.2.0-670-gadc19233falseOK43505636mirror.1trial13020.2.0-670-gadc19233falseOK43575633mirror.2trial13020.2.0-670-gadc19233falseOK43675634mirror.3trial13020.2.0-670-gadc19233trueOKtest2034a92f-64c1-454c-ab67-4a6055d08190up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43675634mirror.3trial1302026-02-18 06:35:27cluster2d076923e-6155-4cb6-a58a-2b79864e9efbup+stoppedlocal image is primary2026-02-18 06:35:27' 2026-02-18T06:35:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:35:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:35:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:35:27' 2026-02-18T06:35:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:35:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:35:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:35:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:35:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:35:31.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test_remove_data_pool 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test_remove_data_pool 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:31.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:35:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:35:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:35:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:35:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:35:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:35:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:35:31.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_remove_data_pool 2026-02-18T06:35:31.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:31.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:31.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:31.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_remove_data_pool ']' 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_remove_data_pool' 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_remove_data_pool 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:35:31.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_remove_data_pool 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:31.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:35:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:35:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:35:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:35:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:35:31.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:31.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:31.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:35:31.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_remove_data_pool 2026-02-18T06:35:31.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_remove_data_pool 2026-02-18T06:35:31.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_remove_data_pool 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool '' 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_remove_data_pool 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2026-02-18T06:35:31.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:31.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_remove_data_pool 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_remove_data_pool 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool '..*' 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_remove_data_pool 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:35:31.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2026-02-18T06:35:31.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:35:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_remove_data_pool.status 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_remove_data_pool 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:35:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:35:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:35:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:31.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:32.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position '' 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_remove_data_pool 2026-02-18T06:35:32.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_remove_data_pool: 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 41c784cb-176d-4e50-a8b9-abed3968ccd2 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:35:25 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:35:32.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:35:27' 2026-02-18T06:35:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:35:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:35:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:35:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:35:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:35:32.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:35:32.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:35:32.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:35:32.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:35:32.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:35:32.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:35:32.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:35:25' 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_remove_data_pool up+replaying '2026-02-18 06:35:25' 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:35:25' 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:35:32.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:35:32.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK443444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}43444348mirror.0trial1302026-02-18 06:35:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:35:27test2b3718e88-35d0-42ca-8167-370fab7f7682up+replayingreplaying, {"bytes_per_second":35.0,"entries_behind_primary":0,"entries_per_second":0.4,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}43444348mirror.0trial1302026-02-18 06:35:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:35:27test46043cca4-bf88-4a33-92cb-fd23ce062877up+replayingreplaying, {"bytes_per_second":35.0,"entries_behind_primary":0,"entries_per_second":0.4,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}43444348mirror.0trial1302026-02-18 06:35:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:35:27test_remove_data_pool41c784cb-176d-4e50-a8b9-abed3968ccd2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:35:25cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:35:27' 2026-02-18T06:35:32.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:35:32.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/last_update' 2026-02-18T06:35:32.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:35:25' 2026-02-18T06:35:32.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/state' 2026-02-18T06:35:32.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:35:32.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:35:32.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:35:32.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:35:32.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:32.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:32.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2026-02-18T06:35:32.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:35:32.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster1 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2026-02-18T06:35:33.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'pool_to_remove' does not exist 2026-02-18T06:35:33.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2026-02-18T06:35:33.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:35:33.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster2 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2026-02-18T06:35:34.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'pool_to_remove' does not exist 2026-02-18T06:35:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test_remove_data_pool 2026-02-18T06:35:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:35:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:34.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test_remove_data_pool 2026-02-18T06:35:34.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:34.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:34.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:34.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_remove_data_pool 2026-02-18T06:35:34.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test_remove_data_pool 2026-02-18T06:35:34.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 100% complete...done. 2026-02-18T06:35:34.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:34.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test_remove_data_pool deleted 2026-02-18T06:35:34.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:34.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:34.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test_remove_data_pool 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_remove_data_pool 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2026-02-18T06:35:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:35:34.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2026-02-18T06:35:34.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:35:34.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:35:34.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:35:34.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test_remove_data_pool deleted '' 2026-02-18T06:35:34.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:35:34.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:35:34.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_remove_data_pool 2026-02-18T06:35:34.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:35:34.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:35:34.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:35:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:35:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test_remove_data_pool 2026-02-18T06:35:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:35:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:35:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_remove_data_pool 2026-02-18T06:35:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2026-02-18T06:35:34.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:35:34.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2026-02-18T06:35:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:35:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:35:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:35:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2026-02-18T06:35:34.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:34.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:34.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:34.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:34.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:34.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:34.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:35:34.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "name": "pool_to_remove/test", 2026-02-18T06:35:34.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Stopped" 2026-02-18T06:35:34.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:35:34.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:34.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2026-02-18T06:35:34.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:35.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:35.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:35:35.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:35.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:35.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:35:35.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:35.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:35.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:35.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:35.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:35.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:35.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:35:35.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "name": "pool_to_remove/test", 2026-02-18T06:35:35.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Stopped" 2026-02-18T06:35:35.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:35:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2026-02-18T06:35:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:35:37.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:37.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:37.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:37.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:37.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:37.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:37.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:37.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:37.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:37.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:37.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:{ 2026-02-18T06:35:37.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "name": "pool_to_remove/test", 2026-02-18T06:35:37.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Stopped" 2026-02-18T06:35:37.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:} 2026-02-18T06:35:37.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:35:37.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2026-02-18T06:35:37.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:35:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:35:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:41.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:41.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:41.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:41.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:41.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:41.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:41.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:41.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:41.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:41.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:41.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:41.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:41.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:41.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:41.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:35:41.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:41.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:41.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:41.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:41.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:41.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:41.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:35:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:35:42.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:42.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:42.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:42.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:42.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:42.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:42.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:35:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:43.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:43.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:43.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:43.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:43.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:43.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:43.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:43.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:35:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:35:43.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:43.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:43.116 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:35:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:43.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:35:43.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:43.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:43.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:35:43.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:43.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:35:43.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:43.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:43.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:43.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:43.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:43.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:43.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:43.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:43.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:35:45.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:45.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:45.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:45.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:45.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:45.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:45.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:45.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:45.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:45.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:45.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:45.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:45.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:45.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:35:45.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:45.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:45.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:45.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:45.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:35:45.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:45.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:45.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:35:45.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:45.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:45.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:45.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:45.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:45.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:45.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:35:49.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:49.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:49.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:49.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:49.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:49.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:49.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:49.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:49.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:49.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:49.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:49.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:49.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:49.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:49.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:49.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:49.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:49.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:49.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:49.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:49.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:49.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:49.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:35:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:49.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:49.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:49.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:35:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:49.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:49.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:35:49.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:49.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:49.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:35:49.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:49.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:49.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:49.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:49.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:49.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:49.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:35:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:35:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:35:57.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:57.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:35:57.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:57.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:57.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:57.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:35:57.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:35:57.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:35:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:35:57.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:57.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:57.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:57.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:57.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:35:57.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:57.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:57.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:35:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:35:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:35:57.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:57.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:57.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:35:57.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:57.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:57.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:57.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:35:57.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:35:57.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:57.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:57.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:35:57.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:57.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:35:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:35:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:35:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:35:57.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:35:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:35:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:35:57.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:35:57.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:57.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:57.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:35:57.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:35:57.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:35:57.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:35:57.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:35:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:35:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:36:05.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:36:05.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:05.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:36:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:36:05.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:05.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:05.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:05.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:06.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:06.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:06.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:06.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:06.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:06.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:06.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:06.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:06.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:06.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:36:06.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:36:06.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:36:06.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:06.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:06.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:06.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:36:06.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:06.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:06.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:06.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:06.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:06.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:36:06.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:36:06.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:06.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:36:06.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:06.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:36:06.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:36:06.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:36:06.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:06.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:06.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:06.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:06.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:06.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:06.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:36:06.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:36:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:36:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:36:14.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:36:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:14.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:14.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:14.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:14.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:14.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:14.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:14.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:14.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:14.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:14.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:14.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:36:14.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:36:14.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:36:14.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:14.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:14.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:14.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:14.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:14.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:36:14.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:14.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:14.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:36:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:36:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:14.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:14.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:14.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:14.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:36:14.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:36:22.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:36:22.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:36:22.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:22.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:36:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:36:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:36:22.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:22.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:22.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:22.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:36:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:36:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:36:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:36:22.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:36:22.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:22.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:22.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:22.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:22.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:22.535 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:22.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:22.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:36:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:36:22.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:36:22.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:22.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:22.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:22.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:22.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:22.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:22.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:22.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:22.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:22.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:22.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:22.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:22.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:22.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:36:22.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:36:22.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:36:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:22.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:22.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:36:22.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:36:30.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:36:30.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:36:30.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:30.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:36:30.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:30.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:36:30.653 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:30.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:36:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:36:30.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:36:30.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:30.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:30.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:30.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:30.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:30.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:30.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:36:30.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:36:30.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:36:30.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:30.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:30.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:36:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:30.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:36:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:30.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:30.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:30.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:30.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:36:30.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:30.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:30.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:36:30.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:36:30.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:36:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:30.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:36:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:36:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:36:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:36:38.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:38.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:36:38.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:38.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:38.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:36:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:36:38.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:36:38.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:38.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:38.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:38.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:38.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:36:38.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:36:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:38.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:38.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:38.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:38.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:38.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:36:38.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:38.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:38.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:38.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:36:38.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:36:38.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:36:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:39.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:39.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:39.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:39.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:39.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:36:39.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:36:39.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:39.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:39.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:36:39.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:36:39.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:39.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:39.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:39.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:39.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:39.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:39.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:39.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:36:39.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:36:47.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:36:47.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:36:47.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:47.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:36:47.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:47.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:36:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:36:47.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:36:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:47.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:47.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:47.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:47.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:36:47.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:36:47.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:36:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:47.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:47.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:47.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:47.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:47.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:36:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:36:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:36:47.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:47.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:47.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:47.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:47.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:47.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:47.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:47.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:47.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:47.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:47.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:36:47.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:36:47.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:36:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:47.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:47.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:47.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:36:47.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:36:55.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:36:55.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:36:55.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:55.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:36:55.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:36:55.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:55.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:55.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:36:55.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:55.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:55.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:36:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:55.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:36:55.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:36:55.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:36:55.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:55.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:55.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:36:55.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:55.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:55.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:36:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:36:55.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:55.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:55.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:36:55.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:55.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:55.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:36:55.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:36:55.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:36:55.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:36:55.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:55.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:55.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:36:55.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:36:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:36:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:36:55.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:36:55.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:36:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 16 2026-02-18T06:37:11.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:11.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:11.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:11.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:11.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:11.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:11.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.608 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:11.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:11.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:11.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:11.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:11.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:11.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:11.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:11.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:11.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:11.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:11.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:11.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:11.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:11.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:11.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:11.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 16 2026-02-18T06:37:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:27.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:27.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:27.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:27.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:27.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:27.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:27.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:27.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:27.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:27.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:27.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:27.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:27.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:27.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:27.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:27.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:27.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:27.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:27.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:27.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:27.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:27.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:27.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:27.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:27.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:27.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:27.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:27.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:27.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:27.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:27.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:27.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:27.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:27.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:27.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:27.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:27.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:27.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster2 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster2 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:37:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:28.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:28.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:28.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:28.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:28.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:28.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:28.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:28.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:28.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:28.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:28.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:28.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:28.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:28.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:28.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:28.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:28.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:28.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:28.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:28.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:28.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:28.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:28.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:28.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:28.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:28.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:28.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:28.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:28.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:28.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:28.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:28.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:28.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:28.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:28.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:28.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:28.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:28.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:37:29.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:29.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:29.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:29.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:29.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:29.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:29.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:29.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:29.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:29.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:29.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:29.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:29.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:29.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:29.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:29.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:29.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:29.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:29.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:29.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:29.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:29.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:29.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:29.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:29.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:29.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:29.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:29.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:29.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:29.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:29.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:29.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:29.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:37:31.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:31.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:31.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:31.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:31.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:31.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:31.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:31.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:31.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:31.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:31.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:31.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:31.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:31.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:31.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:31.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:31.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:31.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:31.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:31.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:31.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:31.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:31.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:31.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:31.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:37:35.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:35.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:35.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:35.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:35.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:35.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:35.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:35.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:35.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:35.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:35.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:35.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:35.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:35.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:35.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:35.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:35.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:35.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:35.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:35.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:35.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:35.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:35.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:35.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:35.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:35.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:35.902 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:37:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:43.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:43.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:43.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:43.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:43.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:43.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:43.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:43.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:43.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:43.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:43.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:43.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:43.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:43.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:43.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:43.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:43.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:44.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:44.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:44.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:44.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:44.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:44.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:44.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:44.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:44.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:44.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:44.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:44.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:44.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:37:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:37:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:37:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:37:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:37:52.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:37:52.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:37:52.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:52.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:52.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:37:52.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:52.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:52.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:52.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:37:52.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:37:52.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:52.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:52.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:37:52.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:37:52.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:37:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:37:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:52.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:52.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:37:52.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:37:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:52.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:37:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:37:52.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:37:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:37:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:52.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:52.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:52.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:37:52.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:37:52.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:37:52.310 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:37:52.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:37:52.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:37:52.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:37:52.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:37:52.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:37:52.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:37:52.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:37:52.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:37:52.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:37:52.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:37:52.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:37:52.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:38:00.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:38:00.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:38:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:38:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:38:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:38:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:38:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:38:00.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:38:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:38:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:38:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:00.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:00.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:00.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:00.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:38:00.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:00.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:38:00.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:00.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:38:00.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:38:00.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:38:00.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:00.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:00.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:00.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:38:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:00.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:38:00.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:38:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:00.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:00.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:00.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:38:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:38:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:38:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:38:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:38:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:38:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:38:08.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:08.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:08.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:08.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:08.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:08.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:38:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:38:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:38:08.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:08.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:08.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:08.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:38:08.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:38:08.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:08.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:08.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:08.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:38:08.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:38:08.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:38:08.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:08.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:08.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:08.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:08.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:08.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:08.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:08.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:08.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:08.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:08.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:08.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:08.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:08.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:08.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:38:08.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:38:08.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:38:08.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:08.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:08.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:08.768 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:08.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:38:08.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:38:16.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:38:16.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:38:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:38:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:38:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:16.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:38:16.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:16.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:38:16.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:38:16.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:38:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:16.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:16.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:16.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:16.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:38:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:38:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:38:16.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:16.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:16.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:16.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:16.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:16.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:38:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:38:16.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:38:16.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:16.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:16.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:16.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:16.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:16.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:38:16.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:38:16.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:38:16.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:16.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:17.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:17.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:17.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:38:17.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:38:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:38:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:38:25.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:38:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:25.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:38:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:38:25.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:25.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:25.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:25.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:25.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:25.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:38:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:38:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:38:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:38:25.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:25.113 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:38:25.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:38:25.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:38:25.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:25.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:25.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:25.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:25.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:25.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:25.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:25.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:25.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:25.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:38:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:38:25.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:38:25.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:25.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:25.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:25.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:25.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:25.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:38:25.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:38:33.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:38:33.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:38:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:38:33.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:38:33.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:33.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:33.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:33.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:38:33.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:38:33.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:38:33.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:33.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:33.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:33.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:33.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:33.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:38:33.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:38:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:38:33.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:38:33.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:33.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:33.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:33.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:33.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:33.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:38:33.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:33.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:38:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:33.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:33.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:33.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:33.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:38:33.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:33.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:38:33.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:38:33.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:33.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:33.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:33.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:33.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:33.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:33.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:33.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:38:33.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:38:41.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:38:41.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:38:41.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:41.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:41.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:38:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:38:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:38:41.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:41.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:41.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:41.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:41.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:41.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:38:41.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:38:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:41.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:41.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:41.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:41.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:41.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:41.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:41.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:41.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:41.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:38:41.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:38:41.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:38:41.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:41.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:41.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:41.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:41.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:41.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:41.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:41.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:41.670 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:41.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:38:41.672 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:38:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:41.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:41.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:41.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:38:41.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 16 2026-02-18T06:38:57.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:38:57.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:38:57.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:57.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:57.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:38:57.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:38:57.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:38:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:38:57.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:38:57.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:38:57.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:38:57.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:57.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:57.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:38:57.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:38:57.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:38:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:38:57.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:57.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:57.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:38:57.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:38:57.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:38:57.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:57.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:57.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:38:57.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:57.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:57.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:38:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:38:57.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:38:57.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:38:57.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:38:57.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:38:57.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:38:57.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:38:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:38:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:38:57.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:38:57.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:38:57.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:38:57.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:38:57.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 16 2026-02-18T06:39:13.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:39:13.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:39:13.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:13.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2026-02-18T06:39:13.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:39:13.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:39:13.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:13.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:13.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:13.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2026-02-18T06:39:14.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:39:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:14.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:39:14.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:39:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:39:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:39:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:39:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:39:14.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2026-02-18T06:39:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:39:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:14.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2026-02-18T06:39:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:14.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:14.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:39:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:39:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:39:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:39:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:39:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:39:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:39:14.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:39:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:39:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:39:14.079 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2026-02-18T06:39:14.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:39:14.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:14.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:39:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:39:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:39:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:39:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:39:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:39:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2026-02-18T06:39:14.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2026-02-18T06:39:14.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:14.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:39:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: snapshot rename' 2026-02-18T06:39:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:39:14.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:39:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:39:14 TEST: snapshot rename 2026-02-18T06:39:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:39:14 TEST: snapshot rename 2026-02-18T06:39:14.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ snap_name=snap_rename 2026-02-18T06:39:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:39:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test2 snap_rename_0 2026-02-18T06:39:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_0 2026-02-18T06:39:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap_rename_0 2026-02-18T06:39:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:39:14.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 1 20 2026-02-18T06:39:14.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 1 - 1 2026-02-18T06:39:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_0 snap_rename_1 2026-02-18T06:39:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_0 2026-02-18T06:39:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_1 2026-02-18T06:39:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_0 mirror/test2@snap_rename_1 2026-02-18T06:39:14.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 2 - 1 2026-02-18T06:39:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_1 snap_rename_2 2026-02-18T06:39:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_1 2026-02-18T06:39:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_2 2026-02-18T06:39:14.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_1 mirror/test2@snap_rename_2 2026-02-18T06:39:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 3 - 1 2026-02-18T06:39:14.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_2 snap_rename_3 2026-02-18T06:39:14.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_2 2026-02-18T06:39:14.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_3 2026-02-18T06:39:14.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_2 mirror/test2@snap_rename_3 2026-02-18T06:39:14.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 4 - 1 2026-02-18T06:39:14.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_3 snap_rename_4 2026-02-18T06:39:14.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_3 2026-02-18T06:39:14.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_4 2026-02-18T06:39:14.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_3 mirror/test2@snap_rename_4 2026-02-18T06:39:14.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 5 - 1 2026-02-18T06:39:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_4 snap_rename_5 2026-02-18T06:39:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_4 2026-02-18T06:39:14.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_5 2026-02-18T06:39:14.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_4 mirror/test2@snap_rename_5 2026-02-18T06:39:14.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 6 - 1 2026-02-18T06:39:14.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_5 snap_rename_6 2026-02-18T06:39:14.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_5 2026-02-18T06:39:14.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_6 2026-02-18T06:39:14.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_5 mirror/test2@snap_rename_6 2026-02-18T06:39:14.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 7 - 1 2026-02-18T06:39:14.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_6 snap_rename_7 2026-02-18T06:39:14.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_6 2026-02-18T06:39:14.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_7 2026-02-18T06:39:14.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_6 mirror/test2@snap_rename_7 2026-02-18T06:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 8 - 1 2026-02-18T06:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_7 snap_rename_8 2026-02-18T06:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_7 2026-02-18T06:39:14.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_8 2026-02-18T06:39:14.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_7 mirror/test2@snap_rename_8 2026-02-18T06:39:14.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 9 - 1 2026-02-18T06:39:14.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_8 snap_rename_9 2026-02-18T06:39:14.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_8 2026-02-18T06:39:14.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_9 2026-02-18T06:39:14.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_8 mirror/test2@snap_rename_9 2026-02-18T06:39:14.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 10 - 1 2026-02-18T06:39:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_9 snap_rename_10 2026-02-18T06:39:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_9 2026-02-18T06:39:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_10 2026-02-18T06:39:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_9 mirror/test2@snap_rename_10 2026-02-18T06:39:14.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 11 - 1 2026-02-18T06:39:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_10 snap_rename_11 2026-02-18T06:39:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_10 2026-02-18T06:39:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_11 2026-02-18T06:39:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_10 mirror/test2@snap_rename_11 2026-02-18T06:39:14.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 12 - 1 2026-02-18T06:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_11 snap_rename_12 2026-02-18T06:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_11 2026-02-18T06:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_12 2026-02-18T06:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_11 mirror/test2@snap_rename_12 2026-02-18T06:39:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 13 - 1 2026-02-18T06:39:14.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_12 snap_rename_13 2026-02-18T06:39:14.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_12 2026-02-18T06:39:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_13 2026-02-18T06:39:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_12 mirror/test2@snap_rename_13 2026-02-18T06:39:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 14 - 1 2026-02-18T06:39:14.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_13 snap_rename_14 2026-02-18T06:39:14.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_13 2026-02-18T06:39:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_14 2026-02-18T06:39:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_13 mirror/test2@snap_rename_14 2026-02-18T06:39:14.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 15 - 1 2026-02-18T06:39:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_14 snap_rename_15 2026-02-18T06:39:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_14 2026-02-18T06:39:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_15 2026-02-18T06:39:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_14 mirror/test2@snap_rename_15 2026-02-18T06:39:14.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 16 - 1 2026-02-18T06:39:14.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_15 snap_rename_16 2026-02-18T06:39:14.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_15 2026-02-18T06:39:14.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_16 2026-02-18T06:39:14.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_15 mirror/test2@snap_rename_16 2026-02-18T06:39:14.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:14.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 17 - 1 2026-02-18T06:39:14.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_16 snap_rename_17 2026-02-18T06:39:14.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:14.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:14.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:14.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_16 2026-02-18T06:39:14.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_17 2026-02-18T06:39:14.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_16 mirror/test2@snap_rename_17 2026-02-18T06:39:15.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:15.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 18 - 1 2026-02-18T06:39:15.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_17 snap_rename_18 2026-02-18T06:39:15.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_17 2026-02-18T06:39:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_18 2026-02-18T06:39:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_17 mirror/test2@snap_rename_18 2026-02-18T06:39:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:15.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 19 - 1 2026-02-18T06:39:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_18 snap_rename_19 2026-02-18T06:39:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_18 2026-02-18T06:39:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_19 2026-02-18T06:39:15.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_18 mirror/test2@snap_rename_19 2026-02-18T06:39:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in `seq 1 20` 2026-02-18T06:39:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ expr 20 - 1 2026-02-18T06:39:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_19 snap_rename_20 2026-02-18T06:39:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap_rename_19 2026-02-18T06:39:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local new_snap=snap_rename_20 2026-02-18T06:39:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_19 mirror/test2@snap_rename_20 2026-02-18T06:39:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:39:15.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test2 snap_rename_20 2026-02-18T06:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap_rename_20 2026-02-18T06:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:39:15.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2026-02-18T06:39:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:39:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:39:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:39:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:39:18.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2026-02-18T06:39:18.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:39:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:39:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:39:18.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:39:22.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2026-02-18T06:39:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:39:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:39:22.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:39:22.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:39:30.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2026-02-18T06:39:30.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:error setting snapshot context: (2) No such file or directory 2026-02-18T06:39:30.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ continue 2026-02-18T06:39:30.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:39:30.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:39:38.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test2': 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 3 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 10fceba6175e 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.10fceba6175e 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:39:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:35:00 2026 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:35:00 2026 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:35:00 2026 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 10fceba6175e 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: b3718e88-35d0-42ca-8167-370fab7f7682 2026-02-18T06:39:38.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:39:38.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:38.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ unprotect_snapshot cluster2 mirror test4 snap1 2026-02-18T06:39:38.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:38.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:38.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:39:38.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:39:38.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap1 2026-02-18T06:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ unprotect_snapshot cluster2 mirror test4 snap2 2026-02-18T06:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap2 2026-02-18T06:39:38.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap2 2026-02-18T06:39:38.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image4} 2026-02-18T06:39:38.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test2 2026-02-18T06:39:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:38.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:38.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test2 2026-02-18T06:39:38.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:38.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:38.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test2 2026-02-18T06:39:38.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test2 2026-02-18T06:39:41.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing all snapshots: 33% complete... Removing all snapshots: 66% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-18T06:39:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test2 2026-02-18T06:39:41.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:39:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for i in ${image2} ${image4} 2026-02-18T06:39:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test4 2026-02-18T06:39:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:39:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test4 2026-02-18T06:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test4 2026-02-18T06:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test4 2026-02-18T06:39:43.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-18T06:39:43.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test4 2026-02-18T06:39:43.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:39:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:39:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: demote image while daemon is offline' 2026-02-18T06:39:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:39:43.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:39:43.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:39:43 TEST: demote image while daemon is offline 2026-02-18T06:39:43.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:39:43 TEST: demote image while daemon is offline 2026-02-18T06:39:43.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image=test_demote_image 2026-02-18T06:39:43.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test_demote_image journal 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test_demote_image 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:43.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:43.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:43.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:43.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:43.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:39:43.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:39:43.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_demote_image 2026-02-18T06:39:43.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:39:43.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:43.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:39:43.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_demote_image 100 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:39:43.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_demote_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:39:43.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:39:43.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:39:43.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_demote_image 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test_demote_image stopped 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:39:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:39:43.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:43.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test_demote_image stopped 2026-02-18T06:39:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:44.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:44.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:39:44.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:39:44.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:44.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:39:44.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:39:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:39:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:39:44.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:39:44.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:44.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:39:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:44.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-18T06:39:44.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:44.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:1 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:1 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:1 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:39:44.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:39:44.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:39:44.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:39:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:39:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:39:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-18T06:39:44.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:44.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:2 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:39:44.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:2 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:39:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:39:44.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:39:44.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:39:44.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:39:44.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-18T06:39:44.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:44.692 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:3 2026-02-18T06:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:39:44.696 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:3 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:3 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:39:44.697 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:39:44.698 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:39:44.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:39:44.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:39:44.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:39:44.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:39:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-18T06:39:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:39:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:39:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:39:45.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:39:45.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:45.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:39:45.755 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:39:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:39:45.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:39:45.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:39:45.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:39:45.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:39:45.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:39:45.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_demote_image 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_demote_image started 2026-02-18T06:39:45.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:39:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:39:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:45.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:46.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_demote_image started 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:46.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:39:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:39:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:39:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:39:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:39:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:46.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:39:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:39:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:39:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_demote_image 2026-02-18T06:39:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:39:46.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:39:46.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:39:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:39:46.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:39:46.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:46.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:46.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:46.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:39:46.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:39:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:39:46.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:39:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:39:46.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:39:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test_demote_image 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test_demote_image 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:39:46.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:39:47.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_demote_image 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_demote_image ']' 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_demote_image' 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_demote_image 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:47.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:47.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:39:47.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:39:47.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:47.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:47.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_demote_image 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:47.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:39:47.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:47.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:39:47.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:47.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:47.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:47.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_demote_image 2026-02-18T06:39:47.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_demote_image 2026-02-18T06:39:47.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_demote_image '' 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:39:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_demote_image 2026-02-18T06:39:47.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:39:47.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:39:47.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:39:47.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:39:47.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2026-02-18T06:39:47.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:39:47.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_demote_image 2026-02-18T06:39:47.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_demote_image '..*' 2026-02-18T06:39:47.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:39:47.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:39:47.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_demote_image 2026-02-18T06:39:47.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:39:47.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:39:47.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:39:47.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:39:47.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2026-02-18T06:39:47.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2026-02-18T06:39:47.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=2, entry_tid=1' 2026-02-18T06:39:47.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' '' 2026-02-18T06:39:47.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:39:47.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = '' 2026-02-18T06:39:47.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2026-02-18T06:39:47.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:39:47.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:39:47.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:39:47.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:39:47.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2026-02-18T06:39:47.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:39:47.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:39:47.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=2 2026-02-18T06:39:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:39:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:39:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:39:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=3 2026-02-18T06:39:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2026-02-18T06:39:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:39:47.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:39:47.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=1 2026-02-18T06:39:47.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 2 -a 3 -le 1 2026-02-18T06:39:47.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:39:47.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:39:47.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:39:47.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_demote_image 2026-02-18T06:39:47.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:47.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:47.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:47.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:39:47.420 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_demote_image ']' 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_demote_image' 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_demote_image 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:39:47.421 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:39:47.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:47.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_demote_image 2026-02-18T06:39:47.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:39:47.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:39:47.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:47.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:39:47.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:39:47.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:47.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:39:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_demote_image 2026-02-18T06:39:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_demote_image 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_demote_image '' 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:39:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_demote_image 2026-02-18T06:39:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:39:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:39:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:39:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:39:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:39:47.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_demote_image 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_demote_image '..*' 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_demote_image 2026-02-18T06:39:47.717 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:39:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:39:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_demote_image 2026-02-18T06:39:47.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:39:47.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:39:47.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:39:47.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:39:47.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:39:47.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+replaying 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:47.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:47.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:47.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+replaying '' '' 2026-02-18T06:39:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:39:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-18T06:39:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:39:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:39:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:39:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:39:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:39:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:46 2026-02-18T06:39:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:39:48.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:39:48.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:39:48.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:39:48.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:47' 2026-02-18T06:39:48.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:39:48.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:39:48.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:39:48.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:39:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:39:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:39:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:39:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:39:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:39:48.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:39:48.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:39:48.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:39:48.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:39:46' 2026-02-18T06:39:48.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+replaying '2026-02-18 06:39:46' 2026-02-18T06:39:48.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:48.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:39:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:39:46' 2026-02-18T06:39:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:48.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:39:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK243444348mirror.0trial13020.2.0-670-gadc19233trueOK43504355mirror.1trial13020.2.0-670-gadc19233falseOK43574365mirror.2trial13020.2.0-670-gadc19233falseOK43674372mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}43444348mirror.0trial1302026-02-18 06:39:22cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:39:47test_demote_imageb2517102-f3ec-4ca3-a07b-15bba59539b2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}43444348mirror.0trial1302026-02-18 06:39:46cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:39:47' 2026-02-18T06:39:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:39:48.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-18T06:39:48.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:39:46' 2026-02-18T06:39:48.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-18T06:39:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:39:48.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:39:48.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:39:48.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+stopped 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:39:48.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:48.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:48.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:48.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+stopped '' '' 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:49.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:49.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:49.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-18T06:39:50.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:47 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:39:50.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:39:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:46' 2026-02-18T06:39:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:39:50.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:39:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:39:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:39:50.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:39:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:39:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:39:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:39:50.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:39:50.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:39:50.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:39:50.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:39:47' 2026-02-18T06:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+stopped '2026-02-18 06:39:47' 2026-02-18T06:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:39:47' 2026-02-18T06:39:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:39:50.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK245714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:39:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2026-02-18 06:39:22test_demote_imageb2517102-f3ec-4ca3-a07b-15bba59539b2up+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:39:47cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2026-02-18 06:39:46' 2026-02-18T06:39:50.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:39:50.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-18T06:39:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:39:47' 2026-02-18T06:39:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-18T06:39:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:39:50.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:39:50.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:39:50.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:39:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster1 2026-02-18T06:39:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:39:50.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:39:50.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:0 '' 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:0 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:0 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:39:50.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:39:50.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:39:50.118 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:39:50.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:39:50.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=33178 2026-02-18T06:39:50.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 33178 ']' 2026-02-18T06:39:50.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 33178 2026-02-18T06:39:50.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:50.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:50.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:50.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33178 '$2 == pid {print; exit 1}' 2026-02-18T06:39:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 33178 0.2 0.0 553232 75328 ? Ssl 06:21 0:03 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:50.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:51.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:51.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33178 '$2 == pid {print; exit 1}' 2026-02-18T06:39:51.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:39:51.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:51.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33178 '$2 == pid {print; exit 1}' 2026-02-18T06:39:51.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:51.160 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:39:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:39:51.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:51.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:39:51.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:0 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:39:51.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:51.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:39:51.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:39:51.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:39:51.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:39:51.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:1 '' 2026-02-18T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:1 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:1 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:39:51.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:39:51.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:39:51.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:39:51.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:39:51.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:39:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:39:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:39:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=33221 2026-02-18T06:39:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 33221 ']' 2026-02-18T06:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 33221 2026-02-18T06:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33221 '$2 == pid {print; exit 1}' 2026-02-18T06:39:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 33221 0.0 0.0 206956 62068 ? Rsl 06:21 0:00 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:52.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:52.226 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33221 '$2 == pid {print; exit 1}' 2026-02-18T06:39:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:39:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:52.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33221 '$2 == pid {print; exit 1}' 2026-02-18T06:39:52.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:39:52.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:39:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:39:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:39:52.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:39:52.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:39:52.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:52.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:39:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:39:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:39:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:39:52.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:39:52.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:39:52.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:39:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:1 2026-02-18T06:39:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:39:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:39:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:39:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:52.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:39:52.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:39:52.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:39:52.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:2 '' 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:2 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:2 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:39:52.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:39:52.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:39:52.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:39:52.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:39:52.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=33283 2026-02-18T06:39:52.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 33283 ']' 2026-02-18T06:39:52.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 33283 2026-02-18T06:39:52.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:52.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:52.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:52.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33283 '$2 == pid {print; exit 1}' 2026-02-18T06:39:52.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 33283 0.0 0.0 338088 58184 ? Ssl 06:21 0:00 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:52.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:52.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:53.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:53.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33283 '$2 == pid {print; exit 1}' 2026-02-18T06:39:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:39:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:53.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33283 '$2 == pid {print; exit 1}' 2026-02-18T06:39:53.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:39:53.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:39:53.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:39:53.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:53.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:39:53.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:39:53.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:39:53.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:39:53.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:53.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:53.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:39:53.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:39:53.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:53.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:53.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:39:53.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:39:53.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:39:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:39:53.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:39:53.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:2 2026-02-18T06:39:53.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:39:53.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:53.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:39:53.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:39:53.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:53.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:39:53.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:39:53.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:39:53.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:39:53.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:53.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:3 '' 2026-02-18T06:39:53.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:39:53.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:39:53.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:53.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:39:53.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:3 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:3 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:39:53.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:39:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:39:53.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:39:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:39:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:39:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=33360 2026-02-18T06:39:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 33360 ']' 2026-02-18T06:39:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 33360 2026-02-18T06:39:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:39:53.390 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33360 '$2 == pid {print; exit 1}' 2026-02-18T06:39:53.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 33360 0.0 0.0 477424 62148 ? Ssl 06:21 0:00 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:53.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:39:53.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:54.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:54.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33360 '$2 == pid {print; exit 1}' 2026-02-18T06:39:54.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:39:54.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:39:54.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=33360 '$2 == pid {print; exit 1}' 2026-02-18T06:39:54.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:39:54.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:54.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:39:54.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:39:54.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:39:54.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:39:54.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:39:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-18T06:39:54.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:39:54.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:39:54.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:39:54.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:39:54.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:39:54.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:39:54.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:3 2026-02-18T06:39:54.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:39:54.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:39:54.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:39:54.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:39:54.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_demote_image 100 2026-02-18T06:39:54.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:54.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:39:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:39:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:39:54.466 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_demote_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:39:54.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:39:54.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:39:54.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7692.45 bytes/sec: 30 MiB/s 2026-02-18T06:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster2 mirror test_demote_image 2026-02-18T06:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test_demote_image 2026-02-18T06:39:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:39:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster1 2026-02-18T06:39:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:54.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:0 2026-02-18T06:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:1 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:54.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:54.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:54.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:2 2026-02-18T06:39:54.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:54.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:39:54.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:39:54.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:54.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:3 2026-02-18T06:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:39:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:39:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:39:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test_demote_image 2026-02-18T06:39:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:39:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:54.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:39:54.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:39:54.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:54.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test_demote_image.status 2026-02-18T06:39:54.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test_demote_image.status 2026-02-18T06:39:54.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:39:54.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:54.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test_demote_image.status 2026-02-18T06:39:54.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:39:54.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test_demote_image.status 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_status_log=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.status 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_demote_image /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.status 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_demote_image --format xml 2026-02-18T06:39:54.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-18T06:39:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:39:54.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-18T06:39:54.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:-:1.1: Document is empty 2026-02-18T06:39:54.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:39:54.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:^ 2026-02-18T06:39:54.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snapshot_id= 2026-02-18T06:39:54.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:39:54.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:39:54.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:39:54.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.status 2026-02-18T06:39:54.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:54.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:54.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:54.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local log=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.status 2026-02-18T06:39:54.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-18T06:39:54.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-18T06:39:54.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:39:54.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-18T06:39:54.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:-:1.1: Document is empty 2026-02-18T06:39:54.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: 2026-02-18T06:39:54.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:^ 2026-02-18T06:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local primary_snapshot_id= 2026-02-18T06:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test '' = '' 2026-02-18T06:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+unknown 2026-02-18T06:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:54.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:39:54.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:54.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:54.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:54.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:55.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+unknown '' '' 2026-02-18T06:39:55.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:55.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:55.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:55.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:39:55.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:55.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:55.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:55.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:55.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-18T06:39:55.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:39:55.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:52 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:39:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:39:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:39:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:55.948 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+unknown '' '' 2026-02-18T06:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-18T06:39:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:39:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:39:57.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:57 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:39:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:39:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:39:57.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:39:57.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:39:57.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:39:57.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:39:57.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:39:57.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:39:57.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:39:57' 2026-02-18T06:39:57.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+unknown '2026-02-18 06:39:57' 2026-02-18T06:39:57.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:39:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:39:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:39:57' 2026-02-18T06:39:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:57.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:39:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING245714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:39:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8down+unknownstatus not foundtest_demote_imageb2517102-f3ec-4ca3-a07b-15bba59539b2up+unknownremote image is not primary45714576mirror.0trial1302026-02-18 06:39:57cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8down+unknownstatus not found' 2026-02-18T06:39:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:39:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-18T06:39:58.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:39:57' 2026-02-18T06:39:58.018 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-18T06:39:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:39:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:39:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:39:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:39:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:39:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+unknown 2026-02-18T06:39:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:39:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:58.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:39:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+unknown '' '' 2026-02-18T06:39:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:39:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:39:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:39:59.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:39:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:39:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:39:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:39:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:39:59.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-18T06:39:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:39:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":101,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:58 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:57' 2026-02-18T06:39:59.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:39:59.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:39:59.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:39:59.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:40:01.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+unknown '' '' 2026-02-18T06:40:01.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:01.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:01.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:40:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:40:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:00 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:40:01.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:40:01.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:40:01.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:39:57' 2026-02-18T06:40:01.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:40:01.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:40:01.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:01.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:40:05.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+unknown '' '' 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-18T06:40:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:04 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:40:05.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:02' 2026-02-18T06:40:05.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+unknown' 2026-02-18T06:40:05.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+unknown 2026-02-18T06:40:05.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:40:05.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: remote image is not primary 2026-02-18T06:40:05.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:05.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:40:05.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:40:05.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:40:05.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:40:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:40:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:40:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:04' 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+unknown '2026-02-18 06:40:04' 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+unknown 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:40:04' 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:05.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:05.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:40:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING1156785682mirror.0trial13020.2.0-670-gadc19233trueOK56845688mirror.1trial13020.2.0-670-gadc19233falseOK56905695mirror.2trial13020.2.0-670-gadc19233falseOK56975702mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}56785682mirror.0trial1302026-02-18 06:39:58cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:40:02test_demote_imageb2517102-f3ec-4ca3-a07b-15bba59539b2up+unknownremote image is not primary56785682mirror.0trial1302026-02-18 06:40:04cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+unknownremote image is not primary2026-02-18 06:40:02' 2026-02-18T06:40:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:40:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-18T06:40:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:04' 2026-02-18T06:40:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-18T06:40:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+unknown 2026-02-18T06:40:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+unknown 2026-02-18T06:40:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+unknown 2026-02-18T06:40:05.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+unknown 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test_demote_image 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:40:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:40:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test_demote_image.export 2026-02-18T06:40:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test_demote_image.export 2026-02-18T06:40:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.export 2026-02-18T06:40:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test_demote_image.export 2026-02-18T06:40:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test_demote_image.export 2026-02-18T06:40:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.export 2026-02-18T06:40:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.export 2026-02-18T06:40:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test_demote_image /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.export 2026-02-18T06:40:05.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:40:05.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test_demote_image /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.export 2026-02-18T06:40:05.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:40:05.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.export 2026-02-18T06:40:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_demote_image.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test_demote_image.export 2026-02-18T06:40:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test_demote_image 2026-02-18T06:40:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force= 2026-02-18T06:40:05.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_demote_image 2026-02-18T06:40:05.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:40:05.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster2 mirror test_demote_image 2026-02-18T06:40:05.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:05.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:05.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:05.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test_demote_image started 2026-02-18T06:40:05.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:05.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:05.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:05.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:40:05.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:05.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:05.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test_demote_image started 2026-02-18T06:40:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:06.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:40:06.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:40:06.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:40:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-18T06:40:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:40:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:40:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:06.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:06.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:40:06.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:40:06.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:06.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:40:06.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:40:06.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:06.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:06.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-18T06:40:06.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:40:06.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:06.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:06.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:40:06.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:06.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:06.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:40:06.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:06.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:06.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:06.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:40:06.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:40:06.541 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:40:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:40:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:40:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = stopped 2026-02-18T06:40:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:40:08.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test_demote_image started 2026-02-18T06:40:08.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:08.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:08.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:08.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:40:08.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:40:08.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:40:08.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:40:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:40:08.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:08.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:40:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:08.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-18T06:40:08.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:40:08.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:40:08.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:40:08.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:40:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:40:08.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:40:08.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:40:08.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:08.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+stopped 2026-02-18T06:40:08.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:08.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:08.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:08.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:40:08.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:40:08.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:08.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:08.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+stopped '' '' 2026-02-18T06:40:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:40:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:40:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-18T06:40:09.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:40:09.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:40:09.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+unknown 2026-02-18T06:40:09.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: remote image is not primary 2026-02-18T06:40:09.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:40:09.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:04 2026-02-18T06:40:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:40:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:40:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:40:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4}} 2026-02-18T06:40:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:07' 2026-02-18T06:40:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:40:09.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:40:09.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:09.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:40:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+stopped '' '' 2026-02-18T06:40:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:40:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:40:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:09 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:40:11.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:40:11.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4}} 2026-02-18T06:40:11.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:07' 2026-02-18T06:40:11.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:40:11.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:40:11.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:40:11.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:40:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:09' 2026-02-18T06:40:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+stopped '2026-02-18 06:40:09' 2026-02-18T06:40:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:11.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:40:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:40:09' 2026-02-18T06:40:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:11.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:40:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK256785682mirror.0trial13020.2.0-670-gadc19233trueOK56845688mirror.1trial13020.2.0-670-gadc19233falseOK56905695mirror.2trial13020.2.0-670-gadc19233falseOK56975702mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}56785682mirror.0trial1302026-02-18 06:39:58cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:40:07test_demote_imageb2517102-f3ec-4ca3-a07b-15bba59539b2up+stoppedlocal image is primary56785682mirror.0trial1302026-02-18 06:40:09cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4}}2026-02-18 06:40:07' 2026-02-18T06:40:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:40:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-18T06:40:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:09' 2026-02-18T06:40:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-18T06:40:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:40:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:40:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:40:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:40:11.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+replaying 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:11.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+replaying '' '' 2026-02-18T06:40:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:40:12.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:12.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:12.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:12.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-18T06:40:12.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_demote_image: 2026-02-18T06:40:12.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b2517102-f3ec-4ca3-a07b-15bba59539b2 2026-02-18T06:40:12.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:40:12.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4}} 2026-02-18T06:40:12.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:40:12.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:12 2026-02-18T06:40:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:40:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:40:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:40:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:40:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:09' 2026-02-18T06:40:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:40:12.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:40:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:40:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4}} 2026-02-18T06:40:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:40:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:40:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:40:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:40:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:40:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:40:12.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:40:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:12' 2026-02-18T06:40:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+replaying '2026-02-18 06:40:12' 2026-02-18T06:40:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:40:12' 2026-02-18T06:40:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:40:12.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK245714576mirror.0trial13020.2.0-670-gadc19233trueOK45784581mirror.1trial13020.2.0-670-gadc19233falseOK45854588mirror.2trial13020.2.0-670-gadc19233falseOK45954603mirror.3trial13020.2.0-670-gadc19233falseOKtestc5801b7e-e96a-45a4-9eeb-d2c6f33ee72aup+stoppedlocal image is primary45714576mirror.0trial1302026-02-18 06:40:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2026-02-18 06:39:58test_demote_imageb2517102-f3ec-4ca3-a07b-15bba59539b2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4}}45714576mirror.0trial1302026-02-18 06:40:12cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:40:09' 2026-02-18T06:40:12.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:40:12.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-18T06:40:12.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:12' 2026-02-18T06:40:12.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-18T06:40:12.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:40:12.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:40:12.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:40:12.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:40:12.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:12.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:12.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster1 mirror test_demote_image 2026-02-18T06:40:12.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:12.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:12.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:12.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster1 mirror test_demote_image 2026-02-18T06:40:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_demote_image 2026-02-18T06:40:12.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_demote_image 2026-02-18T06:40:12.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 rm mirror/test_demote_image 2026-02-18T06:40:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:40:12.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:12.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:40:12.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: disable mirror while daemon is stopped' 2026-02-18T06:40:12.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:40:12.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:40:12.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:40:12 TEST: disable mirror while daemon is stopped 2026-02-18T06:40:12.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:40:12 TEST: disable mirror while daemon is stopped 2026-02-18T06:40:12.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster1 2026-02-18T06:40:12.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:12.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:12.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:0 '' 2026-02-18T06:40:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:0 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:0 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:40:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:40:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:12.900 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:40:12.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:40:12.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:40:12.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=68039 2026-02-18T06:40:12.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 68039 ']' 2026-02-18T06:40:12.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 68039 2026-02-18T06:40:12.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:12.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:12.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:12.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68039 '$2 == pid {print; exit 1}' 2026-02-18T06:40:12.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 68039 0.7 0.0 524012 51860 ? Ssl 06:39 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:12.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:13.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:13.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68039 '$2 == pid {print; exit 1}' 2026-02-18T06:40:13.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:13.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:13.939 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68039 '$2 == pid {print; exit 1}' 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:13.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:13.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:13.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:13.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:13.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:13.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:13.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:13.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:13.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:13.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:13.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:0 2026-02-18T06:40:13.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:13.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:13.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:13.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:40:13.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:40:13.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:1 '' 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:1 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:1 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:40:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:40:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:40:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=68074 2026-02-18T06:40:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 68074 ']' 2026-02-18T06:40:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 68074 2026-02-18T06:40:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:14.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:14.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68074 '$2 == pid {print; exit 1}' 2026-02-18T06:40:14.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 68074 0.4 0.0 223832 42720 ? Ssl 06:39 0:00 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:14.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:14.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68074 '$2 == pid {print; exit 1}' 2026-02-18T06:40:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:15.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68074 '$2 == pid {print; exit 1}' 2026-02-18T06:40:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:40:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:40:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:15.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:15.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:15.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:15.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:15.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:15.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:15.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:15.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:1 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:15.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:15.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:15.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:15.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:40:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:40:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:40:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:2 '' 2026-02-18T06:40:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:40:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:2 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:2 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:40:15.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:40:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:40:15.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:40:15.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=68135 2026-02-18T06:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 68135 ']' 2026-02-18T06:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 68135 2026-02-18T06:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:15.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:15.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68135 '$2 == pid {print; exit 1}' 2026-02-18T06:40:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 68135 0.4 0.0 469712 47344 ? Ssl 06:39 0:00 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68135 '$2 == pid {print; exit 1}' 2026-02-18T06:40:16.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:16.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:16.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68135 '$2 == pid {print; exit 1}' 2026-02-18T06:40:16.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:40:16.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:40:16.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:16.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:16.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:16.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:16.128 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:16.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:16.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:16.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:16.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:16.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:16.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:2 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:40:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:40:16.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:40:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:3 '' 2026-02-18T06:40:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:40:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:16.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:3 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:3 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:40:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:40:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:40:16.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:40:16.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:40:16.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=68197 2026-02-18T06:40:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 68197 ']' 2026-02-18T06:40:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 68197 2026-02-18T06:40:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:16.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68197 '$2 == pid {print; exit 1}' 2026-02-18T06:40:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 68197 0.3 0.0 469708 44352 ? Ssl 06:39 0:00 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:16.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:17.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:17.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68197 '$2 == pid {print; exit 1}' 2026-02-18T06:40:17.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:17.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:17.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=68197 '$2 == pid {print; exit 1}' 2026-02-18T06:40:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:40:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:40:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-18T06:40:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:40:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:17.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:3 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:17.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:17.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:17.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:17.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:17.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:40:17.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:40:17.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:40:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster2 2026-02-18T06:40:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:17.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:17.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:0 '' 2026-02-18T06:40:17.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:0 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:0 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:17.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:17.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:40:17.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:40:17.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:17.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:17.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-18T06:40:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:40:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:40:17.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=37123 2026-02-18T06:40:17.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 37123 ']' 2026-02-18T06:40:17.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 37123 2026-02-18T06:40:17.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:17.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:17.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:17.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37123 '$2 == pid {print; exit 1}' 2026-02-18T06:40:17.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 37123 0.1 0.0 547088 70152 ? Ssl 06:23 0:01 rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:17.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:17.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37123 '$2 == pid {print; exit 1}' 2026-02-18T06:40:18.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:18.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:18.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37123 '$2 == pid {print; exit 1}' 2026-02-18T06:40:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster1 2026-02-18T06:40:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:40:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:40:18.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:18.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:18.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:40:18.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:18.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:40:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:0 2026-02-18T06:40:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:40:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:40:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:18.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-18T06:40:18.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:40:18.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:1 '' 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:1 2026-02-18T06:40:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:1 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:1 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-18T06:40:18.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:40:18.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=37167 2026-02-18T06:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 37167 ']' 2026-02-18T06:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 37167 2026-02-18T06:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:18.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37167 '$2 == pid {print; exit 1}' 2026-02-18T06:40:18.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 37167 0.0 0.0 206952 61980 ? Rsl 06:23 0:00 rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:18.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:18.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:19.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:19.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37167 '$2 == pid {print; exit 1}' 2026-02-18T06:40:19.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:19.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:19.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37167 '$2 == pid {print; exit 1}' 2026-02-18T06:40:19.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster1 2026-02-18T06:40:19.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:40:19.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:19.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:40:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:40:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:40:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:40:19.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:19.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:19.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:19.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:19.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:40:19.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:40:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:40:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:1 2026-02-18T06:40:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:1 2026-02-18T06:40:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:40:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:40:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-18T06:40:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:40:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:40:19.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:19.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:2 '' 2026-02-18T06:40:19.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:40:19.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:19.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:19.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:19.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-18T06:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:40:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:40:19.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=37228 2026-02-18T06:40:19.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 37228 ']' 2026-02-18T06:40:19.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 37228 2026-02-18T06:40:19.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:19.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:19.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:19.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37228 '$2 == pid {print; exit 1}' 2026-02-18T06:40:19.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 37228 0.0 0.0 370880 57968 ? Rsl 06:23 0:00 rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:19.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:19.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:20.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:20.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37228 '$2 == pid {print; exit 1}' 2026-02-18T06:40:20.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:20.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:20.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37228 '$2 == pid {print; exit 1}' 2026-02-18T06:40:20.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster1 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:20.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:40:20.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:20.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:20.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:20.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:20.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:40:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:40:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:20.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:20.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:40:20.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:20.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:20.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:20.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:20.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:40:20.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:40:20.504 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:40:20.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:2 2026-02-18T06:40:20.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:2 2026-02-18T06:40:20.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:20.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:20.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-18T06:40:20.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:40:20.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:3 '' 2026-02-18T06:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:40:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:3 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:3 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:3 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-18T06:40:20.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:40:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:40:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:20.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-18T06:40:20.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:40:20.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:40:20.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=37305 2026-02-18T06:40:20.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 37305 ']' 2026-02-18T06:40:20.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 37305 2026-02-18T06:40:20.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:20.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:20.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:20.534 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37305 '$2 == pid {print; exit 1}' 2026-02-18T06:40:20.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 37305 0.0 0.0 362672 58672 ? Rsl 06:23 0:00 rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:20.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:20.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:21.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37305 '$2 == pid {print; exit 1}' 2026-02-18T06:40:21.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:40:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=37305 '$2 == pid {print; exit 1}' 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster1 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:40:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:40:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:40:21.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:40:21.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:40:21.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:40:21.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:21.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:40:21.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:21.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:21.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:40:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:40:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:40:21.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:3 2026-02-18T06:40:21.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:3 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:40:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-18T06:40:21.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:40:21.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:40:21.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:40:21.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_pool_mirror_mode cluster2 mirror image 2026-02-18T06:40:21.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:21.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:21.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=image 2026-02-18T06:40:21.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2026-02-18T06:40:21.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:note: changing mirroring mode from pool to image 2026-02-18T06:40:21.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ disable_mirror cluster2 mirror test 2026-02-18T06:40:21.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:21.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:21.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:21.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test 2026-02-18T06:40:21.664 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Mirroring disabled 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc1647017e 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc1647017e 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 10fc1647017e 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster1 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:21.699 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:0 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:21.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:21.703 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:21.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:21.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:1 2026-02-18T06:40:21.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:40:21.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:21.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:21.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:21.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:2 2026-02-18T06:40:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:40:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:40:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:21.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:21.745 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:21.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:21.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:3 2026-02-18T06:40:21.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:21.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:40:21.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:21.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:21.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test deleted 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:21.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:21.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:21.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id=10fc1647017e 2026-02-18T06:40:21.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:21.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:40:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 10fc1647017e 2026-02-18T06:40:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:40:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc1647017e 2026-02-18T06:40:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:21.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:21.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc1647017e 2026-02-18T06:40:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc1647017e 2026-02-18T06:40:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 10fc1647017e -o 10fc1647017e = 10fc1647017e 2026-02-18T06:40:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:40:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:40:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:22.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 10fc1647017e 2026-02-18T06:40:22.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:22.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc1647017e 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:22.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc1647017e 2026-02-18T06:40:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc1647017e 2026-02-18T06:40:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 10fc1647017e -o 10fc1647017e = 10fc1647017e 2026-02-18T06:40:22.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:40:22.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:40:22.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:22.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 10fc1647017e 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc1647017e 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:24.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=10fc1647017e 2026-02-18T06:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 10fc1647017e 2026-02-18T06:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 10fc1647017e -o 10fc1647017e = 10fc1647017e 2026-02-18T06:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:40:29.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 10fc1647017e 2026-02-18T06:40:29.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:29.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:29.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:29.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=10fc1647017e 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:29.017 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:29.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2026-02-18T06:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=pool 2026-02-18T06:40:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2026-02-18T06:40:29.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:note: changing mirroring mode from image to pool 2026-02-18T06:40:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ enable_journaling cluster2 mirror test 2026-02-18T06:40:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:29.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 feature enable mirror/test journaling 2026-02-18T06:40:29.103 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:29.104 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:29.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:29.105 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:29.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:40:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:40:29.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-18T06:40:29.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:29.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:29.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:29.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:29.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:40:29.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:29.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:40:29.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:29.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:30.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-18T06:40:30.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:30.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:30.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:30.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:40:30.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:30.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:30.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:30.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:30.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:30.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:30.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:30.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:30.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:30.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:40:30.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:30.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:40:30.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:30.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:32.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:32.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:32.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:40:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:40:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:32.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:40:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-18T06:40:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:40:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:40:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=167aafd34efa 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167aafd34efa 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 167aafd34efa 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:37.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:40:37.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:40:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:40:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:40:37.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:40:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:40:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:37.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:37.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-18T06:40:37.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:37.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:37.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:37.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-18T06:40:37.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:40:37.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:37.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:37.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:37.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-18T06:40:37.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:37.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:37.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:37.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:37.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:37.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-18T06:40:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:40:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:37.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:40:37.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:37.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:37.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:37.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:37.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:37.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-18T06:40:37.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-18T06:40:37.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:37.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:40:38.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:38.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:38.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:38.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:40:38.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:38.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:38.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:38.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:38.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:38.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:38.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:38.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:38.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:38.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:38.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:38.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:38.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:38.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:40:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:40:38.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:40:38.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:40:38.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:40:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:40:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:40:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:40:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: non-default namespace image mirroring' 2026-02-18T06:40:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:40:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:40:38.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:40:38 TEST: non-default namespace image mirroring 2026-02-18T06:40:38.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:40:38 TEST: non-default namespace image mirroring 2026-02-18T06:40:38.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - replay' 2026-02-18T06:40:38.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:40:38.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:40:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:40:38 - replay 2026-02-18T06:40:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:40:38 - replay 2026-02-18T06:40:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns1 test journal 2026-02-18T06:40:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror/ns1 test 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:40:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns1/test 2026-02-18T06:40:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror/ns1 2026-02-18T06:40:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:40:38.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns2 test journal 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror/ns2 test 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:38.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns2/test 2026-02-18T06:40:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror/ns2 2026-02-18T06:40:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ enable_mirror cluster2 mirror/ns2 test journal 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns2/test journal 2026-02-18T06:40:38.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Mirroring enabled 2026-02-18T06:40:38.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 info mirror/ns2/test 2026-02-18T06:40:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test': 2026-02-18T06:40:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 128 MiB in 32 objects 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 0 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 18a5a443b487 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.18a5a443b487 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:40:38 2026 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:40:38 2026 2026-02-18T06:40:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:40:38 2026 2026-02-18T06:40:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 18a5a443b487 2026-02-18T06:40:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:40:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:40:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: d75bf9ca-490e-471d-b075-b0c1e8ba6d77 2026-02-18T06:40:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: true 2026-02-18T06:40:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test 2026-02-18T06:40:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test started 2026-02-18T06:40:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:40:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:38.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:39.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror/ns1 test started 2026-02-18T06:40:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:40:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:40:39.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:40:39.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test 2026-02-18T06:40:39.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:40:39.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-18T06:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:39.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:39.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:39.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:39.975 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:39.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:39.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:39.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:39.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:40.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:40.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:40.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:40.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:40.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:40.082 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:40.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:40.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:40.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:40.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:40.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:40.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:40.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:40.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test 2026-02-18T06:40:40.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:40:40.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:40.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:40:40.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:40.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:40.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:40.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:40.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:40.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:40.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:40:41.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:41.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:41.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:41.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:41.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:41.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:41.256 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:41.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:41.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:41.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:41.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:41.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:41.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:41.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:41.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:41.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:41.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:41.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:41.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:41.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:41.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:41.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:41.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:41.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:41.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:41.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:41.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:41.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:40:41.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:41.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:41.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:40:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:41.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:41.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:41.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:41.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:41.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2026-02-18T06:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:41.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:41.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:40:43.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:43.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-18T06:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:43.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:43.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:43.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:43.433 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:43.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-18T06:40:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:40:43.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:40:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:40:43.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:40:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror/ns2 test 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror/ns2 test started 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:44.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror/ns2 test started 2026-02-18T06:40:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:40:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:40:44.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:40:44.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns2/test 2026-02-18T06:40:44.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:40:44.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:40:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:40:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:40:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:40:44.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns2/test 2026-02-18T06:40:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:44.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:44.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:44.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns2/test 2026-02-18T06:40:44.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:40:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:40:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:40:44.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:40:44.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:40:44.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror/ns1 test 100 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:40:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:40:44.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:40:44.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:40:44.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror/ns2 test 100 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:44.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:40:44.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/ns2/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:40:44.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:40:44.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:40:44.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:40:44.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 mirror/ns1 test 2026-02-18T06:40:44.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 mirror/ns1 test 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:40:44.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:40:44.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:40:44.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:40:44.667 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:40:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test 2026-02-18T06:40:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:44.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2026-02-18T06:40:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2026-02-18T06:40:44.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2026-02-18T06:40:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:44.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:40:44.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:40:44.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:40:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:44.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:44.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2026-02-18T06:40:44.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:44.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:44.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:44.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2026-02-18T06:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:44.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:44.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:40:44.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:44.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:40:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2026-02-18T06:40:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2026-02-18T06:40:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:45.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:40:45.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:45.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:40:45.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:40:45.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:45.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:40:45.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:40:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:45.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:40:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:40:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:45.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=3 2026-02-18T06:40:45.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 1 -a 101 -le 3 2026-02-18T06:40:45.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:40:45.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:40:45.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:40:45.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test 2026-02-18T06:40:45.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:45.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:45.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:45.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:40:45.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:45.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:45.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:45.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2026-02-18T06:40:45.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2026-02-18T06:40:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2026-02-18T06:40:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:40:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:40:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:40:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:40:45.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:40:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:45.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:45.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:45.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:45.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:40:45.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.334 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:45.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:40:45.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:40:45.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:40:45.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:45.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:45.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:40:45.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:40:45.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:45.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:45.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:45.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:40:45.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:45.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:45.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:45.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:45.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2026-02-18T06:40:45.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:45.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2026-02-18T06:40:45.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2026-02-18T06:40:45.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:40:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:40:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:45.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:45.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:45.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:45.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:45.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2026-02-18T06:40:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test.status 2026-02-18T06:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:45.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2026-02-18T06:40:45.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:45.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:45.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.status 2026-02-18T06:40:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:45.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns2 mirror/ns2 test 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns2 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns2 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns2 mirror/ns2 test 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns2 2026-02-18T06:40:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns2 2026-02-18T06:40:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:40:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:40:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:40:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:40:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns2 test 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns2/test' 2026-02-18T06:40:46.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns2/test 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:46.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns2/test 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:46.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:46.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:46.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns2/test 2026-02-18T06:40:46.132 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns2 test 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns2 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns2 test '' 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns2 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns2-test.status 2026-02-18T06:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns2-test.status 2026-02-18T06:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.status 2026-02-18T06:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test 2026-02-18T06:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.status 2026-02-18T06:40:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.status 2026-02-18T06:40:46.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns2 test 2026-02-18T06:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns2 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns2 test '..*' 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns2 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns2-test.status 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns2-test.status 2026-02-18T06:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.status 2026-02-18T06:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.status 2026-02-18T06:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test 2026-02-18T06:40:46.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:46.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:46.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:46.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:46.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:46.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.status 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns1 test 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position '' 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:47.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:47.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test 2026-02-18T06:40:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:40:47.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 851aa6eb-e41b-4583-be3b-bbf6d7750805 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:41 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:40:47.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:40:47.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:40:47.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:40:47.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:40:47.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:47.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:40:47.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:40:47.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:40:47.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:40:47.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:40:47.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:40:47.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:41' 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test up+replaying '2026-02-18 06:40:41' 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:40:41' 2026-02-18T06:40:47.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:47.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:47.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2026-02-18T06:40:47.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING157495754mirror.0trial13020.2.0-670-gadc19233trueOK57565760mirror.1trial13020.2.0-670-gadc19233falseOK57625766mirror.2trial13020.2.0-670-gadc19233falseOK57715780mirror.3trial13020.2.0-670-gadc19233falseOKtest851aa6eb-e41b-4583-be3b-bbf6d7750805up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57495754mirror.0trial1302026-02-18 06:40:41cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:40:47.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:40:47.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:40:47.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:41' 2026-02-18T06:40:47.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:40:47.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:40:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:40:47.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:40:47.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns2 test 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:47.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:40:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position '' 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:48.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:48.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns2/test 2026-02-18T06:40:48.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:40:48.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: d75bf9ca-490e-471d-b075-b0c1e8ba6d77 2026-02-18T06:40:48.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:40:48.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:40:48.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:42 2026-02-18T06:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:40:48.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:40:48.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:40:48.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:40:48.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:48.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:40:48.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:40:48.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:40:48.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:40:48.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:40:48.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:42' 2026-02-18T06:40:48.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns2 test up+replaying '2026-02-18 06:40:42' 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:40:42' 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns2 --verbose --format xml 2026-02-18T06:40:48.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING157495754mirror.0trial13020.2.0-670-gadc19233trueOK57565760mirror.1trial13020.2.0-670-gadc19233falseOK57625766mirror.2trial13020.2.0-670-gadc19233falseOK57715780mirror.3trial13020.2.0-670-gadc19233falseOKtestd75bf9ca-490e-471d-b075-b0c1e8ba6d77up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57495754mirror.0trial1302026-02-18 06:40:42cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:40:48.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:40:48.353 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:40:48.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:42' 2026-02-18T06:40:48.354 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:40:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:40:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:40:48.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:40:48.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:40:48.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:48.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:48.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror/ns1 mirror/ns1 test 2026-02-18T06:40:48.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:40:48.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:48.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:48.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:40:48.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:40:48.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:48.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror/ns1-test.export 2026-02-18T06:40:48.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror/ns1-test.export 2026-02-18T06:40:48.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:48.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export 2026-02-18T06:40:48.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror/ns1-test.export 2026-02-18T06:40:48.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror/ns1-test.export 2026-02-18T06:40:48.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:48.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:40:48.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:40:48.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/ns1/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export 2026-02-18T06:40:48.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:40:48.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/ns1/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:40:48.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:40:48.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test.export 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror/ns2 mirror/ns2 test 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns2 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns2 2026-02-18T06:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror/ns2-test.export 2026-02-18T06:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror/ns2-test.export 2026-02-18T06:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.export 2026-02-18T06:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror/ns2-test.export 2026-02-18T06:40:48.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror/ns2-test.export 2026-02-18T06:40:48.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:48.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns2-test.export 2026-02-18T06:40:48.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns2-test.export 2026-02-18T06:40:48.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/ns2/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.export 2026-02-18T06:40:48.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:40:48.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/ns2/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns2-test.export 2026-02-18T06:40:48.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:40:49.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns2-test.export 2026-02-18T06:40:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns2-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns2-test.export 2026-02-18T06:40:49.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:49.085 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - disable mirroring / delete image' 2026-02-18T06:40:49.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:40:49.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:40:49.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:40:49 - disable mirroring / delete image 2026-02-18T06:40:49.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:40:49 - disable mirroring / delete image 2026-02-18T06:40:49.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror/ns1 test 2026-02-18T06:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:49.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror/ns1 test 2026-02-18T06:40:49.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:49.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:49.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test 2026-02-18T06:40:49.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test 2026-02-18T06:40:49.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:40:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ disable_mirror cluster2 mirror/ns2 test 2026-02-18T06:40:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/ns2/test 2026-02-18T06:40:49.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Mirroring disabled 2026-02-18T06:40:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror/ns1 test deleted 2026-02-18T06:40:49.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns1 test 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:49.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2026-02-18T06:40:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:49.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:40:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:40:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror/ns1 test deleted '' 2026-02-18T06:40:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:40:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:40:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns1 test 2026-02-18T06:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2026-02-18T06:40:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:49.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:49.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:40:49.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror/ns2 test deleted 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns2 test 2026-02-18T06:40:49.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:49.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns2 2026-02-18T06:40:49.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:49.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2026-02-18T06:40:49.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:49.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:49.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:40:49.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:40:49.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror/ns2 test deleted '' 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:40:49.701 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns2 test 2026-02-18T06:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns2 2026-02-18T06:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2026-02-18T06:40:49.702 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:40:49.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:40:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror/ns2 test 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:40:49.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror/ns2 test 2026-02-18T06:40:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns2 2026-02-18T06:40:49.728 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns2/test 2026-02-18T06:40:49.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/ns2/test 2026-02-18T06:40:49.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: mirror to a different remote namespace' 2026-02-18T06:40:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:40:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:40:49.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:40:49 TEST: mirror to a different remote namespace 2026-02-18T06:40:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:40:49 TEST: mirror to a different remote namespace 2026-02-18T06:40:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - replay' 2026-02-18T06:40:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:40:49.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:40:49.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:40:49 - replay 2026-02-18T06:40:49.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:40:49 - replay 2026-02-18T06:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ NS3=ns3 2026-02-18T06:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ NS4=ns4 2026-02-18T06:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 namespace create mirror/ns3 2026-02-18T06:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 namespace create mirror/ns4 2026-02-18T06:40:49.859 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns3 pool --remote-namespace ns4 2026-02-18T06:40:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns4 pool --remote-namespace ns3 2026-02-18T06:40:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns4 test journal 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns4 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror/ns4 test 2026-02-18T06:40:49.915 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns4 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:40:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns4/test 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror/ns4 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns4 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror/ns3 test 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror/ns3 test started 2026-02-18T06:40:50.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:50.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:40:50.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:50.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:40:50.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:40:50.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:50.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:40:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror/ns3 test started 2026-02-18T06:40:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:51.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:40:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:40:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:40:51.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:40:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:40:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:51.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:51.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:51.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:51.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:51.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:51.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:51.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:51.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:51.207 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:51.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:51.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:51.212 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:51.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:51.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:51.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:51.261 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:51.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:51.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:40:51.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:51.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:51.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:51.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:51.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:51.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:51.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:51.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:51.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:51.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:40:51.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:51.320 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:51.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:51.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:51.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:51.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:51.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:51.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:51.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:51.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:51.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:51.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:51.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:40:52.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:52.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:52.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:52.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:52.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:52.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:52.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:52.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:52.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:52.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:52.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:52.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:52.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.431 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:52.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:52.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:52.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:52.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:40:52.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:52.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:52.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:52.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:52.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:52.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:52.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:52.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:52.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:40:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:52.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:52.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:52.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:52.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:52.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:52.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:52.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:52.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:52.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:52.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:52.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:52.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:52.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:52.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:52.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:52.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:52.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:52.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:52.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:52.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:52.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:52.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:40:54.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:54.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:54.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:54.599 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:54.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:54.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:54.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:54.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:54.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:54.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:54.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:54.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:54.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:54.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:40:54.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:40:54.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:54.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:54.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:40:54.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:40:54.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:40:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:54.665 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:54.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:40:54.666 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:54.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:54.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:40:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:40:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:40:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:40:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:40:54.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:54.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:54.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:40:54.720 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:54.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:54.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:54.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:54.764 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:40:54.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:40:54.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:54.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:54.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:40:54.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:40:54.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:40:54.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:54.774 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:54.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:40:54.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:54.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:40:54.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:40:54.815 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:40:54.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:54.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:40:58.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:58.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:58.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:40:58.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:40:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:58.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:58.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:58.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:58.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:40:58.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:40:58.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:40:58.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:40:58.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:40:58.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:40:58.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:40:58.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:40:58.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:58.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror/ns4 test 100 2026-02-18T06:40:58.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:58.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns4 2026-02-18T06:40:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:40:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:40:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:40:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:40:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/ns4/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:40:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:40:58.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:40:58.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:40:58.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns3 mirror/ns4 test 2026-02-18T06:40:58.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns3 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns4 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns3 mirror/ns4 test 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns3 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns4 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:40:58.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:40:58.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:40:58.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:40:58.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns3 test 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns3/test' 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns3/test 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns3/test 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:40:59.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns3/test 2026-02-18T06:40:59.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:59.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns4 test 2026-02-18T06:40:59.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns4 test '' 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:59.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns4/test 2026-02-18T06:40:59.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:59.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:59.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:59.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:59.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:40:59.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:59.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns4 test 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns4 test '..*' 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:59.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns4/test 2026-02-18T06:40:59.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:59.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:59.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:59.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:59.263 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:40:59.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:40:59.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:59.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:40:59.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:40:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:40:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:40:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:59.270 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:40:59.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:40:59.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:40:59.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:40:59.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=3 2026-02-18T06:40:59.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 1 -a 101 -le 3 2026-02-18T06:40:59.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:40:59.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:40:59.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:40:59.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns3 test 2026-02-18T06:40:59.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns3/test' 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns3/test 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:40:59.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:40:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:40:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:40:59.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:40:59.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns3/test 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:40:59.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:40:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:40:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:40:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:40:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:40:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:40:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:40:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:40:59.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:59.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:59.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:40:59.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns3/test 2026-02-18T06:40:59.537 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:59.538 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns4 test 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns4 test '' 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:59.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns4/test 2026-02-18T06:40:59.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:59.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:59.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:59.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:59.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns4 test 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns4 test '..*' 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns4 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns4-test.status 2026-02-18T06:40:59.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:40:59.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns4/test 2026-02-18T06:40:59.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:40:59.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:40:59.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:40:59.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:59.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:40:59.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.status 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns3 test 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns3 test up+replaying primary_position 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:40:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:00.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror/ns3 test up+replaying primary_position '' 2026-02-18T06:41:00.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:00.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:00.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:00.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:00.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:00.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:00.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:00.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:00.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns3/test 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 81607ea4-7450-4f17-9af7-746b4ce54a63 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:40:55 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:41:00.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:41:00.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:41:00.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:41:00.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:41:00.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:41:00.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:00.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:41:00.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:55' 2026-02-18T06:41:00.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns3 test up+replaying '2026-02-18 06:40:55' 2026-02-18T06:41:00.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:00.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:00.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:40:55' 2026-02-18T06:41:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:00.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns3 --verbose --format xml 2026-02-18T06:41:00.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING157495754mirror.0trial13020.2.0-670-gadc19233trueOK57565760mirror.1trial13020.2.0-670-gadc19233falseOK57625766mirror.2trial13020.2.0-670-gadc19233falseOK57715780mirror.3trial13020.2.0-670-gadc19233falseOKtest81607ea4-7450-4f17-9af7-746b4ce54a63up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57495754mirror.0trial1302026-02-18 06:40:55cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:41:00.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:41:00.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:41:00.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:40:55' 2026-02-18T06:41:00.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:41:00.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:41:00.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:41:00.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:00.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror/ns3 mirror/ns4 test 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns3 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns4 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror/ns4-test.export 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror/ns4-test.export 2026-02-18T06:41:00.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:00.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.export 2026-02-18T06:41:00.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror/ns3-test.export 2026-02-18T06:41:00.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror/ns3-test.export 2026-02-18T06:41:00.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:00.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:00.688 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:00.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/ns4/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.export 2026-02-18T06:41:00.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:00.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/ns3/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:00.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns4-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - disable mirroring and re-enable without remote-namespace' 2026-02-18T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:00 - disable mirroring and re-enable without remote-namespace 2026-02-18T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:00 - disable mirroring and re-enable without remote-namespace 2026-02-18T06:41:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror/ns4 test 2026-02-18T06:41:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns4 2026-02-18T06:41:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror/ns4 test 2026-02-18T06:41:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns4 2026-02-18T06:41:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:00.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns4/test 2026-02-18T06:41:00.986 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/ns4/test 2026-02-18T06:41:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:41:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror/ns3 test deleted 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns3 test 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns3/test 2026-02-18T06:41:01.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:01.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:41:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror/ns3 test deleted '' 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns3 test 2026-02-18T06:41:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns3/test 2026-02-18T06:41:01.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:01.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:41:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:01.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool disable mirror/ns3 2026-02-18T06:41:01.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool disable mirror/ns4 2026-02-18T06:41:01.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 namespace create mirror/ns3 2026-02-18T06:41:01.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns3 pool 2026-02-18T06:41:01.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns3 pool 2026-02-18T06:41:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns3 test journal 2026-02-18T06:41:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:01.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror/ns3 test 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:01.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:41:01.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:01.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns3/test 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror/ns3 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror/ns3 test 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:01.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror/ns3 test started 2026-02-18T06:41:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:41:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:02.769 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror/ns3 test started 2026-02-18T06:41:02.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:02.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:02.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:02.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:41:02.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:02.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:02.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:02.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:02.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:02.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:02.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:02.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:02.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:02.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:02.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:02.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:02.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:02.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:02.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:02.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:02.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:02.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:02.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:02.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:02.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:02.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:02.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:02.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:02.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:02.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:02.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:02.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:02.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:02.943 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:02.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:02.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:02.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:02.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:02.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:41:03.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:03.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:03.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:03.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:41:03.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:03.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:04.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:04.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:04.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:04.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:04.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:04.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:04.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:04.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:04.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:04.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns3/test 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:04.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:04.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:04.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:04.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:04.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:04.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:04.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:04.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns3/test 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:04.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:04.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:04.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:04.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:04.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:04.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:04.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:04.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:04.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns3/test 2026-02-18T06:41:04.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:04.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:41:06.216 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:06.217 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:06.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:06.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:06.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:06.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:06.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:06.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:06.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:06.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:06.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:06.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:06.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:06.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:06.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:06.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:06.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:06.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:06.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:06.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:06.339 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:06.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:06.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:06.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:06.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:06.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:06.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:06.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:06.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:06.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns3/test'] 2026-02-18T06:41:06.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:06.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:06.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:06.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:10.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:10.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns3/test 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:10.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:10.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns3/test 2026-02-18T06:41:10.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:10.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:41:10.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:41:10.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:41:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:41:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:41:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:10.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror/ns3 test 100 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:41:10.503 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/ns3/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:41:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:41:10.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:41:10.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 9091.08 bytes/sec: 36 MiB/s 2026-02-18T06:41:10.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns3 mirror/ns3 test 2026-02-18T06:41:10.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:10.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:10.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns3 2026-02-18T06:41:10.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns3 2026-02-18T06:41:10.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns3 mirror/ns3 test 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns3 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns3 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:41:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:41:10.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns3 test 2026-02-18T06:41:10.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:10.753 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns3/test' 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns3/test 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns3/test 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:10.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:10.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:10.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:10.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:10.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:10.770 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns3/test 2026-02-18T06:41:10.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns3 test 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns3 test '' 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:10.816 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:41:10.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns3-test.status 2026-02-18T06:41:10.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns3-test.status 2026-02-18T06:41:10.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:10.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:10.817 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns3/test 2026-02-18T06:41:10.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:10.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:10.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:10.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[]], state=connected] 2026-02-18T06:41:10.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:10.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:10.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns3 test 2026-02-18T06:41:10.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:10.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:10.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:10.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns3 test '..*' 2026-02-18T06:41:10.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:10.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:10.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:10.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:41:10.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns3-test.status 2026-02-18T06:41:10.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns3-test.status 2026-02-18T06:41:10.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:10.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:10.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns3/test 2026-02-18T06:41:10.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:10.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[]], state=connected] 2026-02-18T06:41:10.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:10.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos= 2026-02-18T06:41:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = '' 2026-02-18T06:41:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test '' '!=' '' 2026-02-18T06:41:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:41:10.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.4 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns3 test 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test ']' 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns3/test' 2026-02-18T06:41:11.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns3/test 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns3/test 2026-02-18T06:41:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:11.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:11.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:11.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:11.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:11.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:11.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:11.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:11.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns3/test 2026-02-18T06:41:11.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:11.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns3 test 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns3 test '' 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns3-test.status 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns3-test.status 2026-02-18T06:41:11.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:11.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:11.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns3/test 2026-02-18T06:41:11.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:11.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:11.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:11.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:11.378 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns3 test 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns3 test '..*' 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns3-test.status 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns3-test.status 2026-02-18T06:41:11.379 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:11.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:11.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns3/test 2026-02-18T06:41:11.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:11.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.status 2026-02-18T06:41:11.411 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns3 test 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns3 test up+replaying primary_position 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:11.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:11.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:11.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:11.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:11.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:12.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror/ns3 test up+replaying primary_position '' 2026-02-18T06:41:12.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:12.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns3/test 2026-02-18T06:41:12.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: b43a3640-7193-456b-9c10-74c5e2800008 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:41:10 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:41:12.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:41:12.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:41:12.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:41:12.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:41:12.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:41:12.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:12.447 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:41:12.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:41:12.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:41:12.449 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:41:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:10' 2026-02-18T06:41:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns3 test up+replaying '2026-02-18 06:41:10' 2026-02-18T06:41:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:12.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:41:10' 2026-02-18T06:41:12.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:12.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:12.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns3 --verbose --format xml 2026-02-18T06:41:12.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING157495754mirror.0trial13020.2.0-670-gadc19233trueOK57565760mirror.1trial13020.2.0-670-gadc19233falseOK57625766mirror.2trial13020.2.0-670-gadc19233falseOK57715780mirror.3trial13020.2.0-670-gadc19233falseOKtestb43a3640-7193-456b-9c10-74c5e2800008up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57495754mirror.0trial1302026-02-18 06:41:10cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:41:12.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:41:12.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:41:12.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:10' 2026-02-18T06:41:12.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:41:12.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:41:12.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:12.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror/ns3 mirror/ns3 test 2026-02-18T06:41:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:41:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:12.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:12.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns3 2026-02-18T06:41:12.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns3 2026-02-18T06:41:12.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:12.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror/ns3-test.export 2026-02-18T06:41:12.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:12.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror/ns3-test.export 2026-02-18T06:41:12.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.export 2026-02-18T06:41:12.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror/ns3-test.export 2026-02-18T06:41:12.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror/ns3-test.export 2026-02-18T06:41:12.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:12.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:12.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:12.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/ns3/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.export 2026-02-18T06:41:12.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:12.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/ns3/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:12.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:12.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns3-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns3-test.export 2026-02-18T06:41:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror/ns3 test 2026-02-18T06:41:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:12.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror/ns3 test 2026-02-18T06:41:12.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:12.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns3/test 2026-02-18T06:41:12.790 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/ns3/test 2026-02-18T06:41:12.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror/ns3 test deleted 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:12.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns3 test 2026-02-18T06:41:12.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:12.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:12.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:12.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:12.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns3/test 2026-02-18T06:41:12.874 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror/ns3 test deleted '' 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns3 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror/ns3 test 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns3 2026-02-18T06:41:12.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:12.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/ns3/test 2026-02-18T06:41:12.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:13.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:41:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 mirror pool disable mirror/ns3 2026-02-18T06:41:13.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 mirror pool disable mirror/ns3 2026-02-18T06:41:13.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - data pool' 2026-02-18T06:41:13.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:13.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:13.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:13 - data pool 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:13 - data pool 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ dp_image=test_data_pool 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns1 test_data_pool journal 128 --data-pool mirror_parent 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror/ns1 test_data_pool 128 --data-pool mirror_parent 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 128 ']' 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=128 2026-02-18T06:41:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:13.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/ns1/test_data_pool 2026-02-18T06:41:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror/ns1 2026-02-18T06:41:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_data_pool cluster2 mirror/ns1 test_data_pool 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 info mirror/ns1/test_data_pool 2026-02-18T06:41:13.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-18T06:41:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ data_pool=mirror_parent 2026-02-18T06:41:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test mirror_parent = mirror_parent 2026-02-18T06:41:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test_data_pool 2026-02-18T06:41:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:13.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test_data_pool started 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:14.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror/ns1 test_data_pool started 2026-02-18T06:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:14.125 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:14.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:14.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:14.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:14.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:14.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:14.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:14.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:14.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:14.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:14.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:14.189 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:14.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:14.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:14.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:14.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:14.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:14.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:14.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:14.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:14.253 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:14.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:14.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:14.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:14.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:14.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:14.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:14.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:14.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:14.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:14.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:14.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:41:15.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:15.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:15.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:15.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:15.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:15.412 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:15.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:15.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:15.467 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:15.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:15.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:15.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:15.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:15.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:15.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:15.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:15.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:15.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:15.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:15.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:15.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:15.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:15.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:15.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:15.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-18T06:41:15.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:15.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:15.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:15.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:41:17.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:17.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:17.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:17.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:17.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:17.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:17.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:17.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:17.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:17.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:17.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:17.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:17.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:17.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-18T06:41:17.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:17.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:41:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:41:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:17.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:41:17.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:41:17.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:41:17.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:17.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_data_pool cluster1 mirror/ns1 test_data_pool 2026-02-18T06:41:17.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:17.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:17.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:17.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 info mirror/ns1/test_data_pool 2026-02-18T06:41:17.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-18T06:41:17.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ data_pool=mirror_parent 2026-02-18T06:41:17.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test mirror_parent = mirror_parent 2026-02-18T06:41:17.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror/ns1 test_data_pool 100 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:41:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:41:17.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:41:17.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:41:17.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:41:17.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 mirror/ns1 test_data_pool 2026-02-18T06:41:17.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:17.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:17.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:41:17.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:41:17.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 mirror/ns1 test_data_pool 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:41:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:41:17.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:41:17.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:41:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_data_pool ']' 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test_data_pool' 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2026-02-18T06:41:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2026-02-18T06:41:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:17.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:17.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:17.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:17.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2026-02-18T06:41:17.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:17.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2026-02-18T06:41:17.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '' 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2026-02-18T06:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:18.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:18.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:18.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:18.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:41:18.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.029 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:18.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2026-02-18T06:41:18.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:18.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:18.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:18.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '..*' 2026-02-18T06:41:18.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:18.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:18.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:18.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:41:18.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:18.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:18.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2026-02-18T06:41:18.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:18.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:18.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2026-02-18T06:41:18.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:41:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:41:18.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=2 2026-02-18T06:41:18.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:41:18.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:41:18.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:41:18.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=1 2026-02-18T06:41:18.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:18.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:41:18.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:41:18.068 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=101 2026-02-18T06:41:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2026-02-18T06:41:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:41:18.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:41:18.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=3 2026-02-18T06:41:18.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 2 = 1 -a 101 -le 3 2026-02-18T06:41:18.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:41:18.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:41:18.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:41:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2026-02-18T06:41:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_data_pool ']' 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/ns1/test_data_pool' 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:18.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:18.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:18.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:18.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:18.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:18.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:18.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:18.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:18.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:18.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:18.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:18.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2026-02-18T06:41:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '' 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:18.335 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:41:18.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:18.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:18.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:18.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2026-02-18T06:41:18.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:18.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:18.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:18.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:18.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:18.366 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:18.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2026-02-18T06:41:18.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '..*' 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror/ns1 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_data_pool 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2026-02-18T06:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2026-02-18T06:41:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2026-02-18T06:41:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.status 2026-02-18T06:41:18.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns1 test_data_pool 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position '' 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test_data_pool 2026-02-18T06:41:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_data_pool: 2026-02-18T06:41:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: d805f012-ae8b-4cd7-8d3e-da691c51ef76 2026-02-18T06:41:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:41:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:41:16 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:41:19.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:41:19.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:41:19.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:41:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:41:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:41:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:41:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:41:19.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:41:19.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:16' 2026-02-18T06:41:19.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test_data_pool up+replaying '2026-02-18 06:41:16' 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:41:16' 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:19.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2026-02-18T06:41:19.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING157495754mirror.0trial13020.2.0-670-gadc19233trueOK57565760mirror.1trial13020.2.0-670-gadc19233falseOK57625766mirror.2trial13020.2.0-670-gadc19233falseOK57715780mirror.3trial13020.2.0-670-gadc19233falseOKtest_data_poold805f012-ae8b-4cd7-8d3e-da691c51ef76up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57495754mirror.0trial1302026-02-18 06:41:16cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:41:19.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:41:19.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2026-02-18T06:41:19.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:16' 2026-02-18T06:41:19.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2026-02-18T06:41:19.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:41:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:19.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:41:19.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror/ns1 mirror/ns1 test_data_pool 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror/ns1 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror/ns1 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror/ns1-test_data_pool.export 2026-02-18T06:41:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror/ns1-test_data_pool.export 2026-02-18T06:41:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.export 2026-02-18T06:41:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror/ns1-test_data_pool.export 2026-02-18T06:41:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror/ns1-test_data_pool.export 2026-02-18T06:41:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test_data_pool.export 2026-02-18T06:41:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test_data_pool.export 2026-02-18T06:41:19.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/ns1/test_data_pool /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.export 2026-02-18T06:41:19.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:19.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/ns1/test_data_pool /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test_data_pool.export 2026-02-18T06:41:19.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:19.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test_data_pool.export 2026-02-18T06:41:19.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror_ns1-test_data_pool.export 2026-02-18T06:41:19.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:19.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror/ns1 test_data_pool 2026-02-18T06:41:19.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:19.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror/ns1 test_data_pool 2026-02-18T06:41:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror/ns1 2026-02-18T06:41:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_data_pool 2026-02-18T06:41:19.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test_data_pool 2026-02-18T06:41:19.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test_data_pool 2026-02-18T06:41:19.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:41:19.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:19.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: simple image resync' 2026-02-18T06:41:19.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:19.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:19 TEST: simple image resync 2026-02-18T06:41:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:19 TEST: simple image resync 2026-02-18T06:41:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ request_resync_image cluster1 mirror test image_id 2026-02-18T06:41:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id_var_name=image_id 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:20.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'image_id='\''167aafd34efa'\''' 2026-02-18T06:41:20.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ image_id=167aafd34efa 2026-02-18T06:41:20.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'test -n "$image_id"' 2026-02-18T06:41:20.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -n 167aafd34efa 2026-02-18T06:41:20.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2026-02-18T06:41:20.034 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Flagged image for resync from primary 2026-02-18T06:41:20.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test deleted 167aafd34efa 2026-02-18T06:41:20.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:20.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:20.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:20.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:41:20.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167aafd34efa 2026-02-18T06:41:20.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:20.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167aafd34efa 2026-02-18T06:41:20.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:20.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 167aafd34efa 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167aafd34efa 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:20.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:20.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:20.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:20.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:41:20.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:20.190 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:20.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:20.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:20.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:20.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:20.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:20.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:20.314 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:20.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:20.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:20.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:41:20.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:20.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:21.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-18T06:41:21.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:21.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:21.319 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:21.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:41:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:41:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-18T06:41:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=167a2511f8d 2026-02-18T06:41:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167a2511f8d 2026-02-18T06:41:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 167a2511f8d 2026-02-18T06:41:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:41:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:24.380 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:24.382 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:24.383 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:24.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:41:24.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:24.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:41:24.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:41:24.443 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:24.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:41:24.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:41:24.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:41:24.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:41:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:25.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:41:25.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:41:25.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 15c9d7e5-ef10-49d4-b9c5-a1cb315e3cf2 2026-02-18T06:41:25.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:41:25.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:41:23 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:41:25.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:41:25.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:41:25.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:41:25.482 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:41:25.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:25.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:25.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:41:25.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:25.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:41:25.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:41:25.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:41:25.485 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:41:25.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:23' 2026-02-18T06:41:25.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:41:23' 2026-02-18T06:41:25.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:25.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:25.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:25.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:25.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:41:23' 2026-02-18T06:41:25.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:25.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:25.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:41:25.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING157495754mirror.0trial13020.2.0-670-gadc19233trueOK57565760mirror.1trial13020.2.0-670-gadc19233falseOK57625766mirror.2trial13020.2.0-670-gadc19233falseOK57715780mirror.3trial13020.2.0-670-gadc19233falseOKtest15c9d7e5-ef10-49d4-b9c5-a1cb315e3cf2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}57495754mirror.0trial1302026-02-18 06:41:23cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:41:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:41:25.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:41:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:23' 2026-02-18T06:41:25.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:41:25.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:41:25.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:25.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:41:25.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:25.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:25.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:41:25.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:41:25.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:41:25.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:25.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:25.527 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:25.528 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:41:25.618 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:25.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:25.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:25.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:25.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:25.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:25.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:41:25.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: image resync while replayer is stopped' 2026-02-18T06:41:25.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:25.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:25 TEST: image resync while replayer is stopped 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:25 TEST: image resync while replayer is stopped 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/test 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:25.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:25.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/test 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:25.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:25.812 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:25.813 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/test 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:41:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:26.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-18T06:41:26.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:26.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:26.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:26.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:26.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:26.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:26.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:26.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:41:26.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:26.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:41:26.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:41:26.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:26.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:41:26.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:26.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ request_resync_image cluster1 mirror test image_id 2026-02-18T06:41:26.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id_var_name=image_id 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:26.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'image_id='\''167a2511f8d'\''' 2026-02-18T06:41:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ image_id=167a2511f8d 2026-02-18T06:41:26.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'test -n "$image_id"' 2026-02-18T06:41:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -n 167a2511f8d 2026-02-18T06:41:26.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2026-02-18T06:41:26.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Flagged image for resync from primary 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:26.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:26.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:26.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:26.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:26.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:27.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:27.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:27.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:27.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2026-02-18T06:41:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test deleted 167a2511f8d 2026-02-18T06:41:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:41:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167a2511f8d 2026-02-18T06:41:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167a2511f8d 2026-02-18T06:41:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:27.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:27.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 167a2511f8d 2026-02-18T06:41:27.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167a2511f8d 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:27.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:27.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-18T06:41:27.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:27.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:27.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:41:27.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:27.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:27.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:27.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:27.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2026-02-18T06:41:27.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:27.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:27.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:27.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:27.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:27.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2026-02-18T06:41:27.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:27.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-18T06:41:27.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:27.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:27.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id=167ab8f78a7e 2026-02-18T06:41:27.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:27.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present 167ab8f78a7e 2026-02-18T06:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167ab8f78a7e 2026-02-18T06:41:27.376 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:27.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=167ab8f78a7e 2026-02-18T06:41:27.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167ab8f78a7e 2026-02-18T06:41:27.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 167ab8f78a7e -o 167ab8f78a7e = 167ab8f78a7e 2026-02-18T06:41:27.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:27.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:27.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:27.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:28.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:41:28.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:28.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:28.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:28.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:28.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:28.410 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:28.414 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:28.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:28.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:28.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:28.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:41:28.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:28.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:41:28.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:41:28.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:28.470 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:28.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:28.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:28.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:28.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:28.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:28.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:41:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 15c9d7e5-ef10-49d4-b9c5-a1cb315e3cf2 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:41:28 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:41:29.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:41:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:41:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:41:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:41:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:41:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:41:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2026-02-18T06:41:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:41:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:41:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:41:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:41:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:41:29.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:28' 2026-02-18T06:41:29.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:41:28' 2026-02-18T06:41:29.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:29.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:41:28' 2026-02-18T06:41:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:29.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:41:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING157495754mirror.0trial13020.2.0-670-gadc19233trueOK57565760mirror.1trial13020.2.0-670-gadc19233falseOK57625766mirror.2trial13020.2.0-670-gadc19233falseOK57715780mirror.3trial13020.2.0-670-gadc19233falseOKtest15c9d7e5-ef10-49d4-b9c5-a1cb315e3cf2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}57495754mirror.0trial1302026-02-18 06:41:28cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:41:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:41:29.544 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:41:29.545 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:28' 2026-02-18T06:41:29.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:41:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:41:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:41:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:41:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:29.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:41:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:29.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:29.791 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:29.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:29.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: request image resync while daemon is offline' 2026-02-18T06:41:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:29 TEST: request image resync while daemon is offline 2026-02-18T06:41:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:29 TEST: request image resync while daemon is offline 2026-02-18T06:41:29.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster1 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:0 '' 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:0 2026-02-18T06:41:29.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:0 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:41:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:41:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:41:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=69307 2026-02-18T06:41:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 69307 ']' 2026-02-18T06:41:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 69307 2026-02-18T06:41:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:29.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:29.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69307 '$2 == pid {print; exit 1}' 2026-02-18T06:41:29.856 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 69307 0.8 0.0 543480 65428 ? Ssl 06:40 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:41:29.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:29.857 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69307 '$2 == pid {print; exit 1}' 2026-02-18T06:41:30.869 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:41:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69307 '$2 == pid {print; exit 1}' 2026-02-18T06:41:30.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:30.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:30.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:30.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:30.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:30.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:30.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:30.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:30.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-18T06:41:30.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:30.894 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:30.896 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:30.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:30.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:30.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:0 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:41:30.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:41:30.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:41:30.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:1 '' 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:1 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:1 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:41:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:41:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:41:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=69351 2026-02-18T06:41:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 69351 ']' 2026-02-18T06:41:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 69351 2026-02-18T06:41:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:30.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69351 '$2 == pid {print; exit 1}' 2026-02-18T06:41:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 69351 0.1 0.0 0 0 ? Rs 06:40 0:00 [rbd-mirror] 2026-02-18T06:41:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:30.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69351 '$2 == pid {print; exit 1}' 2026-02-18T06:41:31.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:41:31.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:31.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69351 '$2 == pid {print; exit 1}' 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:31.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:31.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:31.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:31.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:31.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:31.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:31.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:1 2026-02-18T06:41:31.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:31.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:31.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:31.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:31.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:41:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:41:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:41:32.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:32.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:2 '' 2026-02-18T06:41:32.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:41:32.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:2 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:2 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:32.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:32.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:41:32.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:41:32.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:32.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:32.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:41:32.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:41:32.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:41:32.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=69411 2026-02-18T06:41:32.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 69411 ']' 2026-02-18T06:41:32.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 69411 2026-02-18T06:41:32.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:32.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:32.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:32.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69411 '$2 == pid {print; exit 1}' 2026-02-18T06:41:32.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 69411 0.2 0.0 297596 46500 ? Ssl 06:40 0:00 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:41:32.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:32.035 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:33.037 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69411 '$2 == pid {print; exit 1}' 2026-02-18T06:41:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:41:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:33.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69411 '$2 == pid {print; exit 1}' 2026-02-18T06:41:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:41:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:33.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-18T06:41:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:41:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:33.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:33.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:2 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:41:33.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:41:33.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:41:33.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:33.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:3 '' 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:3 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:3 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:41:33.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:41:33.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:41:33.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=69480 2026-02-18T06:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 69480 ']' 2026-02-18T06:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 69480 2026-02-18T06:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:33.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69480 '$2 == pid {print; exit 1}' 2026-02-18T06:41:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:ubuntu 69480 0.1 0.0 469716 48536 ? Ssl 06:40 0:00 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:41:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:34.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:34.123 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69480 '$2 == pid {print; exit 1}' 2026-02-18T06:41:34.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:41:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:41:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=69480 '$2 == pid {print; exit 1}' 2026-02-18T06:41:34.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:34.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:41:34.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:34.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:34.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:34.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:34.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:34.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-18T06:41:34.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:41:34.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:34.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:34.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:34.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:34.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:34.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:3 2026-02-18T06:41:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:34.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:34.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:41:34.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:41:34.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:41:34.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ request_resync_image cluster1 mirror test image_id 2026-02-18T06:41:34.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:34.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:34.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id_var_name=image_id 2026-02-18T06:41:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'image_id='\''167ab8f78a7e'\''' 2026-02-18T06:41:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ image_id=167ab8f78a7e 2026-02-18T06:41:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'test -n "$image_id"' 2026-02-18T06:41:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -n 167ab8f78a7e 2026-02-18T06:41:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2026-02-18T06:41:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Flagged image for resync from primary 2026-02-18T06:41:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster1 2026-02-18T06:41:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:34.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:34.246 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:0 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:34.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:34.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:34.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:34.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:41:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:1 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:34.267 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:41:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:34.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:41:34.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:34.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:2 2026-02-18T06:41:34.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:34.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:41:34.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:34.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:34.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:41:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:3 2026-02-18T06:41:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:41:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:34.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test deleted 167ab8f78a7e 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167ab8f78a7e 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167ab8f78a7e 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:34.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 167ab8f78a7e 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167ab8f78a7e 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:34.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=167ab8f78a7e 2026-02-18T06:41:34.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167ab8f78a7e 2026-02-18T06:41:34.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 167ab8f78a7e -o 167ab8f78a7e = 167ab8f78a7e 2026-02-18T06:41:34.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:34.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:41:34.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:34.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 167ab8f78a7e 2026-02-18T06:41:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167ab8f78a7e 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=167ab8f78a7e 2026-02-18T06:41:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167ab8f78a7e 2026-02-18T06:41:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 167ab8f78a7e -o 167ab8f78a7e = 167ab8f78a7e 2026-02-18T06:41:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:41:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:35.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:41:37.490 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 167ab8f78a7e 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167ab8f78a7e 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:37.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:37.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:37.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=167ab8f78a7e 2026-02-18T06:41:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 167ab8f78a7e 2026-02-18T06:41:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 167ab8f78a7e -o 167ab8f78a7e = 167ab8f78a7e 2026-02-18T06:41:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:41:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:37.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:41:41.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test deleted 167ab8f78a7e 2026-02-18T06:41:41.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:41.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:41.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=167ab8f78a7e 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:41.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:41.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=17479bec92b 2026-02-18T06:41:41.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 17479bec92b 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 167ab8f78a7e -o 167ab8f78a7e = 17479bec92b 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:41.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:41.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id=17479bec92b 2026-02-18T06:41:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:41.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror test present 17479bec92b 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=17479bec92b 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror test 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:41.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:41.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test 2026-02-18T06:41:41.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-18T06:41:41.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=17479bec92b 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 17479bec92b 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 17479bec92b -o 17479bec92b = 17479bec92b 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-18T06:41:41.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:41.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:41.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:41.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:41:41.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:41.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:41.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:42.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-18T06:41:42.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:42.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:42.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:42.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:41:42.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:42.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:42.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-18T06:41:42.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:42.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:42.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:42.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:42.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:42.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:42.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:42.935 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-18T06:41:42.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:42.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:41:42.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:41:42.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:41:42.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2026-02-18T06:41:42.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:42.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:42.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:42.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:42.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:42.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:42.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:42.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:43.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2026-02-18T06:41:43.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:43.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:43.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:43.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:41:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test: 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 15c9d7e5-ef10-49d4-b9c5-a1cb315e3cf2 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:41:40 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:41:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:41:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:41:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:41:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:41:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:41:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:41:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2026-02-18T06:41:44.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:44.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:44.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:41:44.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:41:44.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:41:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:41:44.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:41:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:40' 2026-02-18T06:41:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-18 06:41:40' 2026-02-18T06:41:44.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:41:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:41:40' 2026-02-18T06:41:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:44.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:41:44.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKtest15c9d7e5-ef10-49d4-b9c5-a1cb315e3cf2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}59545959mirror.0trial1302026-02-18 06:41:40cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:41:44.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:41:44.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-18T06:41:44.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:40' 2026-02-18T06:41:44.059 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-18T06:41:44.060 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:41:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:41:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:41:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror test 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:41:44.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-test.export 2026-02-18T06:41:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-test.export 2026-02-18T06:41:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:41:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-test.export 2026-02-18T06:41:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-test.export 2026-02-18T06:41:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:44.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:44.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:44.066 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export 2026-02-18T06:41:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:41:44.252 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-test.export 2026-02-18T06:41:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test 2026-02-18T06:41:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:41:44.336 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test 2026-02-18T06:41:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test 2026-02-18T06:41:44.338 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2026-02-18T06:41:44.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test 2026-02-18T06:41:44.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:41:44.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:44.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:44.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: client disconnect' 2026-02-18T06:41:44.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:44.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:44 TEST: client disconnect 2026-02-18T06:41:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:44 TEST: client disconnect 2026-02-18T06:41:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image=laggy 2026-02-18T06:41:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror laggy 128 --journal-object-size 64K 2026-02-18T06:41:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 128 ']' 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=128 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:44.430 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --journal-object-size 64K mirror/laggy 2026-02-18T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror laggy 10 2026-02-18T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=10 2026-02-18T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand 2026-02-18T06:41:44.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern random 2026-02-18T06:41:44.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:41:44.501 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 10 ops/sec: 3333.4 bytes/sec: 13 MiB/s 2026-02-18T06:41:44.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - replay stopped after disconnect' 2026-02-18T06:41:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:44.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:44.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:44 - replay stopped after disconnect 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:44 - replay stopped after disconnect 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:44.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:41:44.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:44.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:44.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy started 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:45.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:45.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:45.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:45.514 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:45.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:45.518 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:41:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:45.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:45.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:45.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:45.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:45.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:45.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:45.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2026-02-18T06:41:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:45.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:45.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/laggy 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:45.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:45.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:45.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:45.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:45.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/laggy 2026-02-18T06:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:45.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:45.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/laggy 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:45.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:45.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:45.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:45.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/laggy 2026-02-18T06:41:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:45.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:45.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:41:46.735 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:46.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:46.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:46.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:46.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:46.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:46.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:46.752 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:41:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:46.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:46.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:41:46.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:41:46.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:41:46.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2026-02-18T06:41:46.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:46.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:46.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/laggy 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:46.852 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:41:46.853 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:41:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:41:46.861 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:46.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:46.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:41:46.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/laggy 2026-02-18T06:41:46.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:46.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:46.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/laggy 2026-02-18T06:41:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:41:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:46.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:41:46.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:46.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:46.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:41:46.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:41:46.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:41:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:41:46.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/laggy 2026-02-18T06:41:46.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2026-02-18T06:41:46.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:41:46.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:41:46.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:46.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:41:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:48.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:41:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:48.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:48.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:48.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:48.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:48.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:41:49.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:41:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:41:49.027 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:41:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:41:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:41:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:41:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:49.030 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:41:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:41:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:41:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:41:49.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:41:49.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:41:49.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:41:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:41:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:41:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:41:49.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:49.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:41:49.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:41:49.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:41:49.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:49.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:41:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:41:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:41:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:41:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:41:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:41:49.323 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:41:49.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:41:49.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:41:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:49.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:41:49.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:41:49.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:41:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:41:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:41:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:49.360 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:41:49.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:49.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:49.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:49.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:41:49.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:41:49.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:41:49.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ disconnect_image cluster2 mirror laggy 2026-02-18T06:41:49.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:41:49.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:49.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:49.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2026-02-18T06:41:49.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:client 30a4ae41-2c9e-49d5-8ef7-84071acefee8 disconnected 2026-02-18T06:41:49.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:41:49.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:41:49.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:41:49.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:41:49.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:41:49.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:41:49.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:41:49.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:41:49.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=disconnected] 2026-02-18T06:41:49.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' 2026-02-18T06:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2026-02-18T06:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:41:50.457 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:50.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:41:50.458 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:41:50.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:41:50.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:50.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:41:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:41:50.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:41:50.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:41:50.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:41:50.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:41:50.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:41:50.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:41:50.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:50.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:50.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:41:50.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:41:50.519 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:41:50.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:41:50.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:41:50.520 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:41:50.521 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=disconnected 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:41:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:51.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=disconnected 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2026-02-18T06:41:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='laggy: 2026-02-18T06:41:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: e504c8ea-4043-42fe-8f53-7e5b9389dea8 2026-02-18T06:41:51.553 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+error 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: disconnected 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:41:49 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:41:51.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+error' 2026-02-18T06:41:51.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+error 2026-02-18T06:41:51.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*disconnected' 2026-02-18T06:41:51.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: disconnected 2026-02-18T06:41:51.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:41:51.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:41:51.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:41:51.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:41:51.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:41:51.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:41:51.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:41:51.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:41:51.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:49' 2026-02-18T06:41:51.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2026-02-18 06:41:49' 2026-02-18T06:41:51.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:51.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:51.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:51.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:41:51.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:41:49' 2026-02-18T06:41:51.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:41:51.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:41:51.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:41:51.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKlaggye504c8ea-4043-42fe-8f53-7e5b9389dea8up+errordisconnected59545959mirror.0trial1302026-02-18 06:41:49cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:41:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:41:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2026-02-18T06:41:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:41:49' 2026-02-18T06:41:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2026-02-18T06:41:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+error 2026-02-18T06:41:51.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:41:51.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+error 2026-02-18T06:41:51.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:41:51.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:51.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:51.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - replay started after resync requested' 2026-02-18T06:41:51.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:41:51.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:41:51.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:41:51 - replay started after resync requested 2026-02-18T06:41:51.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:41:51 - replay started after resync requested 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ request_resync_image cluster1 mirror laggy image_id 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id_var_name=image_id 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:51.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'image_id='\''1747c80ac8bd'\''' 2026-02-18T06:41:51.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ image_id=1747c80ac8bd 2026-02-18T06:41:51.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'test -n "$image_id"' 2026-02-18T06:41:51.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -n 1747c80ac8bd 2026-02-18T06:41:51.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy 2026-02-18T06:41:51.860 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Flagged image for resync from primary 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 1747c80ac8bd 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747c80ac8bd 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747c80ac8bd 2026-02-18T06:41:51.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:51.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:51.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy deleted 1747c80ac8bd 2026-02-18T06:41:51.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747c80ac8bd 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:51.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=1747c80ac8bd 2026-02-18T06:41:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747c80ac8bd 2026-02-18T06:41:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 1747c80ac8bd -o 1747c80ac8bd = 1747c80ac8bd 2026-02-18T06:41:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:41:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:51.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy deleted 1747c80ac8bd 2026-02-18T06:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747c80ac8bd 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:52.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:52.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:53.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=1747c80ac8bd 2026-02-18T06:41:53.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747c80ac8bd 2026-02-18T06:41:53.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 1747c80ac8bd -o 1747c80ac8bd = 1747c80ac8bd 2026-02-18T06:41:53.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:41:53.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:41:53.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:53.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:41:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy deleted 1747c80ac8bd 2026-02-18T06:41:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:41:55.025 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747c80ac8bd 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:55.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror laggy present 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:55.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:41:55.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:41:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:55.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:41:55.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:41:55.179 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:55.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:41:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:41:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:56.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:41:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:41:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:41:58.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:41:58.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:41:58.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:41:58.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:41:58.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:41:58.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:41:58.262 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:02.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=1747734a37dd 2026-02-18T06:42:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747734a37dd 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 1747734a37dd 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:02.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:02.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:02.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:42:02.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:02.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:02.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:03.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy started 2026-02-18T06:42:03.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:42:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:42:03.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:42:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:42:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:42:03.301 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:42:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:42:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:42:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:03.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:03.306 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:42:03.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:42:03.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:42:03.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:42:03.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:03.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:03.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:03.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:03.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:42:03.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:42:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:42:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:03.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:03.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:03.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:42:03.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:03.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:03.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:03.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:03.575 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:03.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:03.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:03.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:03.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:42:03.627 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:03.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:42:03.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:03.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:03.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:42:03.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:42:03.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:03.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:03.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:03.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=3, entry_tid=3' 2026-02-18T06:42:03.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:03.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:03.661 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:03.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:03.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:03.663 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:42:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:42:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:03.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:03.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=3, entry_tid=3' 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' -a 'object_number=3, tag_tid=3, entry_tid=3' = 'object_number=3, tag_tid=3, entry_tid=3' 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:03.695 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:42:03.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:42:03.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:03.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:03.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:03.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:03.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' 2026-02-18T06:42:03.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:03.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:42:03.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:03.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:03.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:03.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:03.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:03.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-laggy.export 2026-02-18T06:42:03.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-laggy.export 2026-02-18T06:42:03.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:03.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export 2026-02-18T06:42:03.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-laggy.export 2026-02-18T06:42:03.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-laggy.export 2026-02-18T06:42:03.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:03.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:03.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export 2026-02-18T06:42:03.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:42:03.996 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:04.048 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:42:04.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:04.133 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:04.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:04.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - disconnected after max_concurrent_object_sets reached' 2026-02-18T06:42:04.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:42:04.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:42:04 - disconnected after max_concurrent_object_sets reached 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:42:04 - disconnected after max_concurrent_object_sets reached 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/laggy 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:04.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:04.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:04.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/laggy 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:04.158 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:04.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:04.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:04.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:04.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/laggy 2026-02-18T06:42:04.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:42:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:42:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:42:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:42:05.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:05.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:42:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:42:05.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:42:05.225 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:05.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:05.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:05.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:05.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:42:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:42:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:42:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:42:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:42:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:42:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:05.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:05.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:42:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:42:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:05.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2026-02-18T06:42:05.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 1 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=1 2026-02-18T06:42:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 1 2026-02-18T06:42:05.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror laggy 20 16384 2026-02-18T06:42:05.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:05.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:05.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:05.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=20 2026-02-18T06:42:05.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=16384 2026-02-18T06:42:05.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 16384 2026-02-18T06:42:05.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand 2026-02-18T06:42:05.389 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 16384 io_threads 1 bytes 327680 pattern random 2026-02-18T06:42:05.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:42:05.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 20 ops/sec: 2500.05 bytes/sec: 39 MiB/s 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror laggy 20 16384 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=20 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=16384 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 16384 2026-02-18T06:42:05.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand 2026-02-18T06:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 16384 io_threads 1 bytes 327680 pattern random 2026-02-18T06:42:05.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:42:05.444 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 20 ops/sec: 2500.05 bytes/sec: 39 MiB/s 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:05.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:05.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:05.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:05.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:05.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:05.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:05.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:05.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:05.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 1 2026-02-18T06:42:05.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:05.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:05.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=6, entry_tid=41], [object_number=12, tag_tid=6, entry_tid=40], [object_number=27, tag_tid=6, entry_tid=39], [object_number=14, tag_tid=6, entry_tid=38]]], state=connected] 2026-02-18T06:42:05.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=disconnected] 2026-02-18T06:42:05.481 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 0 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=0 2026-02-18T06:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 0 2026-02-18T06:42:05.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - replay is still stopped (disconnected) after restart' 2026-02-18T06:42:05.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:42:05.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:42:05.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:42:05 - replay is still stopped '(disconnected)' after restart 2026-02-18T06:42:05.524 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:42:05 - replay is still stopped (disconnected) after restart 2026-02-18T06:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror start mirror/laggy 2026-02-18T06:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:05.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:05.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:05.526 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/laggy 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:05.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:05.530 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:05.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:05.539 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:05.540 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/laggy 2026-02-18T06:42:05.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:05.585 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:05.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:42:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:42:06.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:42:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:06.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:42:06.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:42:06.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:06.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:06.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:06.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:06.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:42:06.647 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:42:06.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:42:06.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:42:06.648 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=disconnected 2026-02-18T06:42:06.650 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:07.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2026-02-18T06:42:07.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:07.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:07.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:07.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:07.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=disconnected 2026-02-18T06:42:07.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:07.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:07.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:07.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2026-02-18T06:42:07.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='laggy: 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: e504c8ea-4043-42fe-8f53-7e5b9389dea8 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+error 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: disconnected 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:05 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+error' 2026-02-18T06:42:07.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+error 2026-02-18T06:42:07.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*disconnected' 2026-02-18T06:42:07.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: disconnected 2026-02-18T06:42:07.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:42:07.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:05' 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2026-02-18 06:42:05' 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:42:05' 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:07.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:42:07.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKlaggye504c8ea-4043-42fe-8f53-7e5b9389dea8up+errordisconnected59545959mirror.0trial1302026-02-18 06:42:05cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:42:07.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:42:07.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2026-02-18T06:42:07.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:05' 2026-02-18T06:42:07.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2026-02-18T06:42:07.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+error 2026-02-18T06:42:07.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:42:07.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+error 2026-02-18T06:42:07.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:42:07.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:07.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:07.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - replay started after resync requested' 2026-02-18T06:42:07.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:42:07.727 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:42:07 - replay started after resync requested 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:42:07 - replay started after resync requested 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ request_resync_image cluster1 mirror laggy image_id 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id_var_name=image_id 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:07.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:07.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'image_id='\''1747734a37dd'\''' 2026-02-18T06:42:07.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ image_id=1747734a37dd 2026-02-18T06:42:07.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'test -n "$image_id"' 2026-02-18T06:42:07.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -n 1747734a37dd 2026-02-18T06:42:07.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy 2026-02-18T06:42:07.786 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Flagged image for resync from primary 2026-02-18T06:42:07.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 1747734a37dd 2026-02-18T06:42:07.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:07.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:42:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747734a37dd 2026-02-18T06:42:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747734a37dd 2026-02-18T06:42:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:07.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy deleted 1747734a37dd 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747734a37dd 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:07.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:07.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:07.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:07.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:07.891 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=1747734a37dd 2026-02-18T06:42:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747734a37dd 2026-02-18T06:42:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 1747734a37dd -o 1747734a37dd = 1747734a37dd 2026-02-18T06:42:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:42:07.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:42:07.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:07.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy deleted 1747734a37dd 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747734a37dd 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:08.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=1747734a37dd 2026-02-18T06:42:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747734a37dd 2026-02-18T06:42:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z 1747734a37dd -o 1747734a37dd = 1747734a37dd 2026-02-18T06:42:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:42:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = present 2026-02-18T06:42:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:08.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:42:10.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy deleted 1747734a37dd 2026-02-18T06:42:10.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:10.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:10.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:10.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=1747734a37dd 2026-02-18T06:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:10.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:10.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:10.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:10.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:10.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:10.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:10.979 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror laggy present 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:11.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:11.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:42:11.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:11.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:42:11.109 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:11.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:11.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:11.136 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:12.139 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:12.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:12.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:12.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:42:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:42:14.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:14.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:14.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:14.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:14.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:14.191 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:14.192 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:42:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:42:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:18.195 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:18.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:18.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:18.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=17474bb94027 2026-02-18T06:42:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 17474bb94027 2026-02-18T06:42:18.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 17474bb94027 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:18.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:42:18.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:18.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:18.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:19.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy started 2026-02-18T06:42:19.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:19.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:19.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:42:19.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:42:19.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:42:19.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:42:19.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:42:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:19.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:42:19.235 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:19.236 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:19.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:19.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:42:19.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:42:19.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:42:19.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:42:19.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:19.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:42:19.289 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:42:19.290 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:42:19.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:42:19.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:42:19.291 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:19.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:42:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:42:19.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:42:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:19.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:19.499 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:19.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:19.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:19.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:19.500 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:19.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:19.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:19.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:19.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:42:19.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:42:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:19.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:19.560 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:19.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:19.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:19.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:19.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:19.589 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.590 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:19.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:19.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:19.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:19.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:19.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:19.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:19.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:19.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:19.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:19.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:19.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:19.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:19.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:19.654 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-laggy.export 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-laggy.export 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export 2026-02-18T06:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-laggy.export 2026-02-18T06:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-laggy.export 2026-02-18T06:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:19.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:19.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export 2026-02-18T06:42:19.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:42:19.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:42:19.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:19.876 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog ' - rbd_mirroring_resync_after_disconnect config option' 2026-02-18T06:42:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:42:19.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:42:19.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:42:19 - rbd_mirroring_resync_after_disconnect config option 2026-02-18T06:42:19.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:42:19 - rbd_mirroring_resync_after_disconnect config option 2026-02-18T06:42:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect true 2026-02-18T06:42:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect 2026-02-18T06:42:19.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=true 2026-02-18T06:42:19.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect true 2026-02-18T06:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:42:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:42:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:42:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:20.147 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:20.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:20.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:20.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:42:20.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:20.150 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:20.151 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:20.152 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:20.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:20.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:20.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:20.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:20.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:20.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:20.163 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:42:20.208 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:20.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:42:20.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:42:20.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.209 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:42:20.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:20.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:20.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:20.210 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:20.211 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:20.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:20.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:20.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2026-02-18T06:42:20.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:20.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.240 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=25, tag_tid=9, entry_tid=1' 2026-02-18T06:42:20.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:20.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:20.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:20.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:20.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2026-02-18T06:42:20.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2026-02-18T06:42:20.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=25, tag_tid=9, entry_tid=1' -a 'object_number=25, tag_tid=9, entry_tid=1' = 'object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=27, tag_tid=8, entry_tid=3' '!=' '' 2026-02-18T06:42:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:42:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=27, tag_tid=8, entry_tid=3' = '' 2026-02-18T06:42:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:20.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=25, tag_tid=9, entry_tid=1' 2026-02-18T06:42:20.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:42:20.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:20.275 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=9 2026-02-18T06:42:20.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:20.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:42:20.276 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:20.277 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=8 2026-02-18T06:42:20.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=25, tag_tid=9, entry_tid=1' 2026-02-18T06:42:20.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:42:20.278 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:20.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=1 2026-02-18T06:42:20.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3' 2026-02-18T06:42:20.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:42:20.280 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=3 2026-02-18T06:42:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 9 = 8 -a 1 -le 3 2026-02-18T06:42:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:42:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:42:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:20.486 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:20.487 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:20.497 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:20.498 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:42:20.546 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2026-02-18T06:42:20.576 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2026-02-18T06:42:20.577 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=25, tag_tid=9, entry_tid=1' 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.580 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:20.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:20.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:20.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:20.581 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:20.582 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2026-02-18T06:42:20.609 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2026-02-18T06:42:20.610 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:20.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=25, tag_tid=9, entry_tid=1' 2026-02-18T06:42:20.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=25, tag_tid=9, entry_tid=1' -a 'object_number=25, tag_tid=9, entry_tid=1' = 'object_number=25, tag_tid=9, entry_tid=1' 2026-02-18T06:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id=17474bb94027 2026-02-18T06:42:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ disconnect_image cluster2 mirror laggy 2026-02-18T06:42:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2026-02-18T06:42:20.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:client 30a4ae41-2c9e-49d5-8ef7-84071acefee8 disconnected 2026-02-18T06:42:20.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 17474bb94027 2026-02-18T06:42:20.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:20.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:20.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=deleted 2026-02-18T06:42:20.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=17474bb94027 2026-02-18T06:42:20.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:20.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 17474bb94027 2026-02-18T06:42:20.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:20.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy deleted 17474bb94027 2026-02-18T06:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.775 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=deleted 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id=17474bb94027 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:20.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test deleted = deleted 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_present cluster1 mirror laggy present 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=present 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:20.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image_id= 2026-02-18T06:42:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.1 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:20.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:20.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:20.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:20.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:42:20.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:20.954 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:21.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:21.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:21.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:21.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:42:21.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:21.982 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:42:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:24.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image laggy: (2) No such file or directory 2026-02-18T06:42:24.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id= 2026-02-18T06:42:24.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:24.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = deleted 2026-02-18T06:42:24.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-18T06:42:24.011 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:42:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_present cluster1 mirror laggy present '' 2026-02-18T06:42:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=present 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id= 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=deleted 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_image_id 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror laggy 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2026-02-18T06:42:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_image_id=1747f88defb8 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 1747f88defb8 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' -o '' = 1747f88defb8 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=present 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test present = present 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy started 2026-02-18T06:42:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:42:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:42:29.052 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:42:29.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:42:29.106 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:42:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:42:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:42:29.107 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:42:29.108 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:42:29.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:42:29.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:42:29.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:29.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:29.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:42:29.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:42:29.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:42:29.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:42:29.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:29.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:29.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:29.318 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:42:29.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:29.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:42:29.373 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:29.374 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:29.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:29.375 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:29.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:29.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:29.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:29.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:29.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:29.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:29.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:29.406 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:29.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:29.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:29.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:29.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:29.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:29.407 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:29.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:29.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' -a 'object_number=27, tag_tid=10, entry_tid=3' = 'object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:29.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:29.468 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:29.469 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ compare_images cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local ret=0 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster2-mirror-laggy.export 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster2-mirror-laggy.export 2026-02-18T06:42:29.472 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local rmt_export=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export 2026-02-18T06:42:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ mkfname cluster1-mirror-laggy.export 2026-02-18T06:42:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo cluster1-mirror-laggy.export 2026-02-18T06:42:29.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:29.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loc_export=/tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:29.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:29.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export 2026-02-18T06:42:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:42:29.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:29.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-18T06:42:29.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmp /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:29.693 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.export /tmp/tmp.mzUeR8FHMy/cluster1-mirror-laggy.export 2026-02-18T06:42:29.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:29.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect false 2026-02-18T06:42:29.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:29.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:29.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect 2026-02-18T06:42:29.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=false 2026-02-18T06:42:29.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect false 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror laggy 2026-02-18T06:42:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:42:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:42:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:29.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:29.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:42:29.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:29.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:29.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:29.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:29.970 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:29.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:42:30.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:42:30.023 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:30.024 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:30.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:30.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:30.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2026-02-18T06:42:30.053 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:30.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=25, tag_tid=11, entry_tid=1' 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:30.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:30.058 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:30.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:30.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:30.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2026-02-18T06:42:30.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2026-02-18T06:42:30.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:30.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=25, tag_tid=11, entry_tid=1' -a 'object_number=25, tag_tid=11, entry_tid=1' = 'object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:30.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=27, tag_tid=10, entry_tid=3' '!=' '' 2026-02-18T06:42:30.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:42:30.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 'object_number=27, tag_tid=10, entry_tid=3' = '' 2026-02-18T06:42:30.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_mirror_pos='object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:30.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=25, tag_tid=11, entry_tid=1' 2026-02-18T06:42:30.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:42:30.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:30.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_tag=11 2026-02-18T06:42:30.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2026-02-18T06:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:30.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_tag=10 2026-02-18T06:42:30.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=25, tag_tid=11, entry_tid=1' 2026-02-18T06:42:30.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:42:30.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:30.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_entry=1 2026-02-18T06:42:30.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2026-02-18T06:42:30.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2026-02-18T06:42:30.094 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cut -d= -f 2 2026-02-18T06:42:30.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_entry=3 2026-02-18T06:42:30.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 11 = 10 -a 1 -le 3 2026-02-18T06:42:30.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:42:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:42:30.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror laggy 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n laggy ']' 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/laggy' 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:30.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:30.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:42:30.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:42:30.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:30.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:30.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:30.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:30.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:30.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:30.312 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:30.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:30.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:30.313 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror laggy 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '' 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.361 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:42:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:30.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:30.363 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:30.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:30.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:30.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2026-02-18T06:42:30.391 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2026-02-18T06:42:30.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=25, tag_tid=11, entry_tid=1' 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:30.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:30.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:30.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:30.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:30.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2026-02-18T06:42:30.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2026-02-18T06:42:30.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=25, tag_tid=11, entry_tid=1' 2026-02-18T06:42:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=25, tag_tid=11, entry_tid=1' -a 'object_number=25, tag_tid=11, entry_tid=1' = 'object_number=25, tag_tid=11, entry_tid=1' 2026-02-18T06:42:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ disconnect_image cluster2 mirror laggy 2026-02-18T06:42:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:30.428 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2026-02-18T06:42:30.456 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:client 30a4ae41-2c9e-49d5-8ef7-84071acefee8 disconnected 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=laggy 2026-02-18T06:42:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:42:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-laggy.status 2026-02-18T06:42:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-laggy.status 2026-02-18T06:42:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:42:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2026-02-18T06:42:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 6 2026-02-18T06:42:30.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 6 2026-02-18T06:42:30.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:42:30.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2026-02-18T06:42:30.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=disconnected] 2026-02-18T06:42:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-laggy.status 2026-02-18T06:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -z '' 2026-02-18T06:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2026-02-18T06:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:31.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2026-02-18T06:42:31.491 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:42:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:31.494 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:42:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2026-02-18T06:42:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2026-02-18T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=disconnected 2026-02-18T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2026-02-18T06:42:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:32.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=disconnected 2026-02-18T06:42:32.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:32.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:32.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:32.559 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2026-02-18T06:42:32.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='laggy: 2026-02-18T06:42:32.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: e504c8ea-4043-42fe-8f53-7e5b9389dea8 2026-02-18T06:42:32.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+error 2026-02-18T06:42:32.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: disconnected 2026-02-18T06:42:32.591 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:30 2026-02-18T06:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+error' 2026-02-18T06:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+error 2026-02-18T06:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*disconnected' 2026-02-18T06:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: disconnected 2026-02-18T06:42:32.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:42:32.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:42:32.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:42:32.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:42:32.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:42:32.595 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:42:32.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:42:32.596 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:42:32.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:30' 2026-02-18T06:42:32.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2026-02-18 06:42:30' 2026-02-18T06:42:32.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:32.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:32.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:32.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:42:30' 2026-02-18T06:42:32.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:32.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:32.598 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:42:32.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKlaggye504c8ea-4043-42fe-8f53-7e5b9389dea8up+errordisconnected59545959mirror.0trial1302026-02-18 06:42:30cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:42:32.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:42:32.630 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2026-02-18T06:42:32.631 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:30' 2026-02-18T06:42:32.632 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2026-02-18T06:42:32.633 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+error 2026-02-18T06:42:32.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:42:32.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+error 2026-02-18T06:42:32.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:42:32.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:32.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:32.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror laggy 2026-02-18T06:42:32.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:32.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:32.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:42:32.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:32.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror laggy 2026-02-18T06:42:32.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:32.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=laggy 2026-02-18T06:42:32.637 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/laggy 2026-02-18T06:42:32.662 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/laggy 2026-02-18T06:42:32.718 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:42:32.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:32.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: split-brain' 2026-02-18T06:42:32.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:42:32.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:42:32.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:42:32 TEST: split-brain 2026-02-18T06:42:32.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:42:32 TEST: split-brain 2026-02-18T06:42:32.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ image=split-brain 2026-02-18T06:42:32.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror split-brain journal 2026-02-18T06:42:32.724 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror split-brain 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:32.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:42:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:42:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:42:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/split-brain 2026-02-18T06:42:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:42:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:42:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:42:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:42:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror split-brain 2026-02-18T06:42:32.756 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:32.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2026-02-18T06:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:33.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image split-brain: (2) No such file or directory 2026-02-18T06:42:33.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status= 2026-02-18T06:42:33.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:42:33.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:42:33.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:33.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:42:35.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2026-02-18T06:42:35.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:35.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:35.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:35.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='split-brain: 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 95efbecc-619e-4ad6-a483-ed140d58bbbf 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:35 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:42:35.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:42:35.825 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2026-02-18T06:42:35.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:42:35.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:42:35.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:42:35.826 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:42:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:42:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:42:35.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:42:35.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:42:35.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:35' 2026-02-18T06:42:35.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2026-02-18 06:42:35' 2026-02-18T06:42:35.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:35.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:35.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:35.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:35.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:42:35' 2026-02-18T06:42:35.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:35.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:35.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:42:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKsplit-brain95efbecc-619e-4ad6-a483-ed140d58bbbfup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}59545959mirror.0trial1302026-02-18 06:42:35cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:42:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:42:35.862 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-18T06:42:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:35' 2026-02-18T06:42:35.864 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-18T06:42:35.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:42:35.865 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:42:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:42:35.866 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:42:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror split-brain --force 2026-02-18T06:42:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:35.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:35.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:35.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force=--force 2026-02-18T06:42:35.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/split-brain --force 2026-02-18T06:42:35.901 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:42:35.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror split-brain 2026-02-18T06:42:35.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:35.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror split-brain stopped 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:36.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror split-brain stopped 2026-02-18T06:42:36.905 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/split-brain 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:42:36.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:36.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:36.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:42:36.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:42:36.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:36.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:42:36.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/split-brain 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:42:36.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:42:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:42:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:42:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:42:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:42:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:42:36.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:36.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:36.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:42:36.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/split-brain 2026-02-18T06:42:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:42:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:42:36.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:42:36.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+stopped 2026-02-18T06:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:42:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:42:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:37.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+stopped '' '' 2026-02-18T06:42:37.968 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:37.969 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:37.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='split-brain: 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 95efbecc-619e-4ad6-a483-ed140d58bbbf 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:35 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:38.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:38.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:38.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:38.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:42:38.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:42:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:42:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: stopped 2026-02-18T06:42:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:42:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:42:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:42:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:42:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:42:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:42:38.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:35' 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+stopped '2026-02-18 06:42:35' 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:42:35' 2026-02-18T06:42:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:38.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:38.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:42:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKsplit-brain95efbecc-619e-4ad6-a483-ed140d58bbbfup+stoppedstopped59545959mirror.0trial1302026-02-18 06:42:35cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:42:38.040 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:42:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-18T06:42:38.041 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:35' 2026-02-18T06:42:38.042 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-18T06:42:38.043 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:42:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:42:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:42:38.044 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster1 mirror split-brain 10 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=10 2026-02-18T06:42:38.046 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:42:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:42:38.047 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 bench mirror/split-brain --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand 2026-02-18T06:42:38.080 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern random 2026-02-18T06:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 10 ops/sec: 3333.4 bytes/sec: 13 MiB/s 2026-02-18T06:42:38.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ demote_image cluster1 mirror split-brain 2026-02-18T06:42:38.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:38.087 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:38.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:38.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/split-brain 2026-02-18T06:42:38.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image demoted to non-primary 2026-02-18T06:42:38.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+error split-brain 2026-02-18T06:42:38.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:38.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:38.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:38.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:38.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=split-brain 2026-02-18T06:42:38.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:38.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:38.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:39.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2026-02-18T06:42:39.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=split-brain 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='split-brain: 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 95efbecc-619e-4ad6-a483-ed140d58bbbf 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: stopped 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:35 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:39.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:39.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:39.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+error' 2026-02-18T06:42:39.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:42:39.165 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:39.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:42:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=split-brain 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:41.167 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:41.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='split-brain: 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 95efbecc-619e-4ad6-a483-ed140d58bbbf 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+error 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: split-brain detected 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:39 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:41.199 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+error' 2026-02-18T06:42:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+error 2026-02-18T06:42:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*split-brain' 2026-02-18T06:42:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: split-brain detected 2026-02-18T06:42:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:42:41.201 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:42:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:42:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:42:41.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:42:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:42:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:42:41.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:42:41.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:39' 2026-02-18T06:42:41.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+error '2026-02-18 06:42:39' 2026-02-18T06:42:41.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:41.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:41.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:41.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+error 2026-02-18T06:42:41.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:42:39' 2026-02-18T06:42:41.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:41.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:41.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:42:41.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKsplit-brain95efbecc-619e-4ad6-a483-ed140d58bbbfup+errorsplit-brain detected59545959mirror.0trial1302026-02-18 06:42:39cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:42:41.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:42:41.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-18T06:42:41.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:39' 2026-02-18T06:42:41.239 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-18T06:42:41.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+error 2026-02-18T06:42:41.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+error 2026-02-18T06:42:41.241 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+error 2026-02-18T06:42:41.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+error 2026-02-18T06:42:41.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:41.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ request_resync_image cluster1 mirror split-brain image_id 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image_id_var_name=image_id 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_image_id cluster1 mirror split-brain 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=split-brain 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster=cluster1 info mirror/split-brain 2026-02-18T06:42:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-18T06:42:41.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'image_id='\''174756195590'\''' 2026-02-18T06:42:41.273 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ image_id=174756195590 2026-02-18T06:42:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval 'test -n "$image_id"' 2026-02-18T06:42:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -n 174756195590 2026-02-18T06:42:41.274 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/split-brain 2026-02-18T06:42:41.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Flagged image for resync from primary 2026-02-18T06:42:41.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror split-brain 2026-02-18T06:42:41.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:41.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:41.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:41.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2026-02-18T06:42:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='split-brain: 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 95efbecc-619e-4ad6-a483-ed140d58bbbf 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+error 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: split-brain detected 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:39 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:42.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:42.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:42.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:42:42.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:42:42.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:42.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:42:44.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2026-02-18T06:42:44.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:44.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:44.367 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image split-brain: (2) No such file or directory 2026-02-18T06:42:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status= 2026-02-18T06:42:44.368 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:42:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:42:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:44.369 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 4 2026-02-18T06:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2026-02-18T06:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd: error opening image split-brain: (2) No such file or directory 2026-02-18T06:42:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status= 2026-02-18T06:42:48.399 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:42:48.400 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:42:48.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:42:48.401 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 8 2026-02-18T06:42:56.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2026-02-18T06:42:56.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:56.402 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:56.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:56.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:42:56.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:42:56.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:56.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:56.403 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-18T06:42:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='split-brain: 2026-02-18T06:42:56.434 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: 95efbecc-619e-4ad6-a483-ed140d58bbbf 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:42:50 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:42:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:42:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:42:56.436 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:42:56.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:42:56.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:42:56.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:42:56.437 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:42:56.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:42:56.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:42:56.438 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:42:56.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:42:56.439 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:42:56.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:50' 2026-02-18T06:42:56.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2026-02-18 06:42:50' 2026-02-18T06:42:56.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:56.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:56.441 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:42:50' 2026-02-18T06:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:42:56.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKsplit-brain95efbecc-619e-4ad6-a483-ed140d58bbbfup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}59545959mirror.0trial1302026-02-18 06:42:50cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:42:56.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:42:56.474 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-18T06:42:56.475 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:42:50' 2026-02-18T06:42:56.476 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-18T06:42:56.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:42:56.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:42:56.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:42:56.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:56.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror split-brain 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror split-brain 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=split-brain 2026-02-18T06:42:56.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/split-brain 2026-02-18T06:42:56.509 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/split-brain 2026-02-18T06:42:56.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-18T06:42:56.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:56.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2026-02-18T06:42:56.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:42:56.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:42:56.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:42:56 TEST: check if removed 'images'\''' OMAP are removed 2026-02-18T06:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:42:56 TEST: check if removed images' OMAP are removed 2026-02-18T06:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster2 2026-02-18T06:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:42:56.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:56.572 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:0 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:0 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:42:56.573 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:42:56.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:56.574 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:42:56.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:56.592 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:1 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:1 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:42:56.593 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:56.594 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:42:56.612 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:2 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:2 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:42:56.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-18T06:42:56.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:42:56.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:42:56.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:42:56.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:42:56.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:56.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:42:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:42:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster2:3 2026-02-18T06:42:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster2:3 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster2 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster2 2026-02-18T06:42:56.635 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:42:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:42:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:42:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:42:56.636 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_in_omap cluster1 mirror 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=status_global 2026-02-18T06:42:56.657 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:56.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=status_global 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:56.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:56.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:56.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:42:56.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c status_global 2026-02-18T06:42:56.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_ 2026-02-18T06:42:56.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:56.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:56.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:56.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2026-02-18T06:42:56.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:56.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:56.685 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:56.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_ 2026-02-18T06:42:56.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-18T06:42:56.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_ 2026-02-18T06:42:56.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:56.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:56.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:56.686 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirror_leader 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_map 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-18T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 0 2026-02-18T06:42:56.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=1 2026-02-18T06:42:56.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:56.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 -eq 0 ']' 2026-02-18T06:42:56.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:56.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:42:57.734 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:57.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:57.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:42:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 1 = 0 2026-02-18T06:42:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=1 2026-02-18T06:42:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 -eq 0 ']' 2026-02-18T06:42:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:57.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 2 2026-02-18T06:42:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-18T06:42:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:42:59.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-18T06:42:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:42:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:42:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_in_omap cluster2 mirror 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:59.787 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=status_global 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:59.788 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=status_global 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c status_global 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:42:59.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:42:59.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:42:59.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:59.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:42:59.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:59.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2026-02-18T06:42:59.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirroring 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_ 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:59.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_ 2026-02-18T06:42:59.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-18T06:42:59.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:59.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirroring 2026-02-18T06:42:59.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-18T06:42:59.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_ 2026-02-18T06:42:59.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local obj_name=rbd_mirror_leader 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local filter=image_map 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-18T06:42:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local filter=image_map 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local obj_name=rbd_mirror_leader 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2026-02-18T06:42:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -c image_map 2026-02-18T06:42:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test 0 = 0 2026-02-18T06:42:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ error_code=0 2026-02-18T06:42:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set -e 2026-02-18T06:42:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:42:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:42:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:42:59.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: no blocklists' 2026-02-18T06:42:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:42:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:42:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:42:59 TEST: no blocklists 2026-02-18T06:42:59.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:42:59 TEST: no blocklists 2026-02-18T06:42:59.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS='--id admin' 2026-02-18T06:42:59.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster1 osd blocklist ls 2026-02-18T06:42:59.851 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -q 'listed 0 entries' 2026-02-18T06:43:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS='--id admin' 2026-02-18T06:43:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster2 osd blocklist ls 2026-02-18T06:43:00.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -q 'listed 0 entries' 2026-02-18T06:43:00.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:43:00.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ testlog 'TEST: force promote with a user snapshot' 2026-02-18T06:43:00.327 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ tee -a /tmp/tmp.mzUeR8FHMy/rbd-mirror.test.log 2026-02-18T06:43:00.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ date '+%F %T' 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo 2026-02-18 06:43:00 TEST: force promote with a user snapshot 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:2026-02-18 06:43:00 TEST: force promote with a user snapshot 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ force_promote_image=test_force_promote_user 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image_and_enable_mirror cluster2 mirror test_force_promote_user journal 10G 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local mode=journal 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n journal ']' 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_image cluster2 mirror test_force_promote_user 10G 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=128 2026-02-18T06:43:00.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 10G ']' 2026-02-18T06:43:00.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=10G 2026-02-18T06:43:00.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:00.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 create --size 10G --image-feature layering,exclusive-lock,journaling mirror/test_force_promote_user 2026-02-18T06:43:00.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ is_pool_mirror_mode_image mirror 2026-02-18T06:43:00.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:00.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' pool = image ']' 2026-02-18T06:43:00.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ case "${pool}" in 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 1 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_force_promote_user 100 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:43:00.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:43:00.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:43:00.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:43:00.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:43:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote_user 2026-02-18T06:43:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-18T06:43:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:00.415 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:43:01.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-18T06:43:01.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:01.417 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:43:01.418 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:43:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:43:01.419 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:43:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:43:01.422 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:01.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:01.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:43:01.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:43:01.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:01.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:01.423 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:43:01.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:01.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:43:01.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:43:01.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:01.424 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:43:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:43:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:43:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:43:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:43:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:01.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:01.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:01.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:43:01.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:43:01.426 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:43:01.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:01.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:01.427 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:43:01.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:43:01.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:43:01.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:43:01.432 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:1 2026-02-18T06:43:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:43:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:1 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:1 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:43:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:43:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:43:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:43:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:01.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:01.529 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:01.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:01.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.531 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:2 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:43:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:43:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:43:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:43:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:01.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:01.584 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:3 2026-02-18T06:43:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:01.587 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:01.588 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:43:01.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:43:01.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:43:01.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:43:01.597 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:01.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:01.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:01.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:43:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:43:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:43:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:02.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:43:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:43:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:43:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote_user 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote_user started 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=started 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:43:03.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_force_promote_user started 2026-02-18T06:43:03.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=started 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:43:03.709 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:43:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:43:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:43:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:43:03.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:03.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:03.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:03.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:03.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:03.713 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:03.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:03.715 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:03.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:03.716 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:03.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:03.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:03.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:03.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:03.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:03.767 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:03.771 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:03.772 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:03.773 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:03.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:03.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:03.781 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:03.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:03.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:03.824 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:03.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:03.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.827 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:03.829 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:03.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:03.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:03.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:03.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:03.830 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:03.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:03.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:03.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:03.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:03.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:03.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:03.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:03.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:03.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:03.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:03.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:03.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:03.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:03.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:03.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:03.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:03.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:43:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:04.936 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:04.940 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:04.953 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:04.998 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Replaying"' 2026-02-18T06:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Replaying"' 2026-02-18T06:43:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:43:05.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: "state": "Replaying" 2026-02-18T06:43:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ current_state=started 2026-02-18T06:43:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test started = started 2026-02-18T06:43:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test_force_promote_user 2026-02-18T06:43:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test_force_promote_user 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_cluster=cluster1 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local local_pool=mirror 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local remote_pool=mirror 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s master_pos mirror_pos last_mirror_pos 2026-02-18T06:43:05.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2026-02-18T06:43:05.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ true 2026-02-18T06:43:05.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-18T06:43:05.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0.2 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ flush cluster1 mirror test_force_promote_user 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'cmd=rbd mirror flush' 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n test_force_promote_user ']' 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cmd='rbd mirror flush mirror/test_force_promote_user' 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote_user 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_instance=cluster1 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance=cluster1 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local loop_instance 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:43:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote_user 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ shift 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:43:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:05.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:05.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:05.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:05.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:05.206 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:05.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:05.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:05.219 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote_user 2026-02-18T06:43:05.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:05.264 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_master_journal_position cluster2 mirror test_force_promote_user 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote_user 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_force_promote_user '' 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote_user 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local id_regexp= 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:43:05.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote_user 2026-02-18T06:43:05.266 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:43:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:43:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:43:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:43:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:43:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote_user 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote_user 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ get_journal_position cluster2 mirror test_force_promote_user '..*' 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local pool=mirror 2026-02-18T06:43:05.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local image=test_force_promote_user 2026-02-18T06:43:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local 'id_regexp=..*' 2026-02-18T06:43:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ mkfname cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ sed -e 's|[/ ]|_|g' 2026-02-18T06:43:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local status_log=/tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote_user 2026-02-18T06:43:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ tee /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:minimum_set: 0 2026-02-18T06:43:05.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:active_set: 0 2026-02-18T06:43:05.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:registered clients: 2026-02-18T06:43:05.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:43:05.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: [id=30a4ae41-2c9e-49d5-8ef7-84071acefee8, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2026-02-18T06:43:05.326 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.mzUeR8FHMy/cluster2-mirror-test_force_promote_user.status 2026-02-18T06:43:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:43:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2026-02-18T06:43:05.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_force_promote_user 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = journal ']' 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote_user up+replaying primary_position 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:43:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:43:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:43:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:43:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote_user up+replaying primary_position '' 2026-02-18T06:43:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:43:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern=primary_position 2026-02-18T06:43:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:43:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:43:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:43:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote_user 2026-02-18T06:43:06.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote_user: 2026-02-18T06:43:06.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: df6ef9ac-725a-40c3-be08-c153dedb65f0 2026-02-18T06:43:06.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:43:06.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:43:04 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: down+unknown 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: status not found 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: ' 2026-02-18T06:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+replaying' 2026-02-18T06:43:06.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+replaying 2026-02-18T06:43:06.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*primary_position' 2026-02-18T06:43:06.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:43:06.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:43:06.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:43:06.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:43:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:43:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:43:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:43:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:43:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:43:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:43:04' 2026-02-18T06:43:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote_user up+replaying '2026-02-18 06:43:04' 2026-02-18T06:43:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+replaying 2026-02-18T06:43:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:43:04' 2026-02-18T06:43:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:43:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:43:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:43:06.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='WARNINGOKWARNING159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOKtest_force_promote_userdf6ef9ac-725a-40c3-be08-c153dedb65f0up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}59545959mirror.0trial1302026-02-18 06:43:04cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932down+unknownstatus not found' 2026-02-18T06:43:06.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:43:06.600 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-18T06:43:06.601 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:43:04' 2026-02-18T06:43:06.602 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-18T06:43:06.603 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+replaying 2026-02-18T06:43:06.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+replaying 2026-02-18T06:43:06.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+replaying 2026-02-18T06:43:06.604 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+replaying 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:43:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:43:07.606 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped '' '' 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:43:07.607 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote_user 2026-02-18T06:43:07.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote_user: 2026-02-18T06:43:07.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: df6ef9ac-725a-40c3-be08-c153dedb65f0 2026-02-18T06:43:07.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:43:07.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:43:06 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+replaying 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:43:04' 2026-02-18T06:43:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:43:07.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:43:07.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:43:07.641 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:43:07.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:43:07.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:43:07.642 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:43:07.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:43:07.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:43:07.643 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:43:07.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:43:07.644 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:43:07.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:43:06' 2026-02-18T06:43:07.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote_user up+stopped '2026-02-18 06:43:06' 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:43:06' 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:43:07.646 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:43:07.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK167336738mirror.0trial13020.2.0-670-gadc19233trueOK67406743mirror.1trial13020.2.0-670-gadc19233falseOK67476750mirror.2trial13020.2.0-670-gadc19233falseOK67576763mirror.3trial13020.2.0-670-gadc19233falseOKtest_force_promote_userdf6ef9ac-725a-40c3-be08-c153dedb65f0up+stoppedlocal image is primary67336738mirror.0trial1302026-02-18 06:43:06cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2026-02-18 06:43:04' 2026-02-18T06:43:07.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:43:07.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-18T06:43:07.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:43:06' 2026-02-18T06:43:07.677 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-18T06:43:07.678 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:43:07.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:43:07.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:43:07.679 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:43:07.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:07.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:07.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_force_promote_user 100 2026-02-18T06:43:07.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:07.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:07.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:07.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:43:07.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:43:07.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:07.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:43:07.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:43:07.714 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:43:07.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:43:07.725 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:43:07.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ create_snapshot cluster2 mirror test_force_promote_user snap1 2026-02-18T06:43:07.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:07.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:07.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:07.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap=snap1 2026-02-18T06:43:07.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 snap create mirror/test_force_promote_user@snap1 2026-02-18T06:43:08.757 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-18T06:43:08.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_force_promote_user 2560 4194304 2026-02-18T06:43:08.762 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:08.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:08.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:08.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=2560 2026-02-18T06:43:08.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size=4194304 2026-02-18T06:43:08.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n 4194304 2026-02-18T06:43:08.763 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4194304 --io-threads 1 --io-total 10737418240 --io-pattern rand 2026-02-18T06:43:08.798 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4194304 io_threads 1 bytes 10737418240 pattern random 2026-02-18T06:43:09.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:43:09.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 1 205 204.979 820 MiB/s 2026-02-18T06:43:10.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 2 418 209.714 839 MiB/s 2026-02-18T06:43:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 3 613 204.739 819 MiB/s 2026-02-18T06:43:12.814 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 4 819 204.391 818 MiB/s 2026-02-18T06:43:13.811 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 5 1026 205.035 820 MiB/s 2026-02-18T06:43:14.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 6 1225 203.922 816 MiB/s 2026-02-18T06:43:15.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 7 1417 199.445 798 MiB/s 2026-02-18T06:43:16.799 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 8 1597 196.843 787 MiB/s 2026-02-18T06:43:17.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 9 1784 193.507 774 MiB/s 2026-02-18T06:43:18.801 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 10 1977 190.585 762 MiB/s 2026-02-18T06:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 11 2164 187.954 752 MiB/s 2026-02-18T06:43:20.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 12 2353 187.541 750 MiB/s 2026-02-18T06:43:21.800 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: 13 2548 190.204 761 MiB/s 2026-02-18T06:43:21.868 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 13 ops: 2560 ops/sec: 195.887 bytes/sec: 784 MiB/s 2026-02-18T06:43:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:43:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_snap_present cluster1 mirror test_force_promote_user snap1 2026-02-18T06:43:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:21.877 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local snap_name=snap1 2026-02-18T06:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-18T06:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:43:22.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 info mirror/test_force_promote_user@snap1 2026-02-18T06:43:22.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:rbd image 'test_force_promote_user': 2026-02-18T06:43:22.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: size 10 GiB in 2560 objects 2026-02-18T06:43:22.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: order 22 (4 MiB objects) 2026-02-18T06:43:22.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: snapshot_count: 1 2026-02-18T06:43:22.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: id: 1747ece9eb34 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: block_name_prefix: rbd_data.1747ece9eb34 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: format: 2 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: features: layering, exclusive-lock, journaling 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: op_features: 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: flags: 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: create_timestamp: Wed Feb 18 06:43:02 2026 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: access_timestamp: Wed Feb 18 06:43:02 2026 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: modify_timestamp: Wed Feb 18 06:43:02 2026 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: protected: False 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: journal: 1747ece9eb34 2026-02-18T06:43:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring state: enabled 2026-02-18T06:43:22.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring mode: journal 2026-02-18T06:43:22.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring global id: df6ef9ac-725a-40c3-be08-c153dedb65f0 2026-02-18T06:43:22.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: mirroring primary: false 2026-02-18T06:43:22.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:22.909 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:43:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster1 -KILL 2026-02-18T06:43:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig=-KILL 2026-02-18T06:43:23.911 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:0 -KILL 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig=-KILL 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:0 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:0 2026-02-18T06:43:23.913 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:23.914 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:43:23.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:43:23.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:43:23.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=73367 2026-02-18T06:43:23.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 73367 ']' 2026-02-18T06:43:23.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill -KILL 73367 2026-02-18T06:43:23.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:43:23.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:43:23.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:23.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73367 '$2 == pid {print; exit 1}' 2026-02-18T06:43:23.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:43:23.944 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:23.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73367 '$2 == pid {print; exit 1}' 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:23.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:23.973 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:23.974 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:23.976 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:23.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:23.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:0 2026-02-18T06:43:23.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:23.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:23.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:23.989 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:23.990 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:43:24.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:43:24.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:43:24.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:24.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:1 -KILL 2026-02-18T06:43:24.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:43:24.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig=-KILL 2026-02-18T06:43:24.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:24.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:1 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:1 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:24.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:43:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:43:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:24.004 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:43:24.014 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:43:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=73402 2026-02-18T06:43:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 73402 ']' 2026-02-18T06:43:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill -KILL 73402 2026-02-18T06:43:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:43:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73402 '$2 == pid {print; exit 1}' 2026-02-18T06:43:24.032 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:43:24.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:24.033 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73402 '$2 == pid {print; exit 1}' 2026-02-18T06:43:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:24.049 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:24.050 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:24.051 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:24.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:24.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:24.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-18T06:43:24.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:43:24.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:24.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:24.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:24.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:1 2026-02-18T06:43:24.076 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:43:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:43:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:2 -KILL 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig=-KILL 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:2 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:2 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:43:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:43:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:43:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=73471 2026-02-18T06:43:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 73471 ']' 2026-02-18T06:43:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill -KILL 73471 2026-02-18T06:43:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:43:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:43:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73471 '$2 == pid {print; exit 1}' 2026-02-18T06:43:24.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:43:24.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:24.117 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73471 '$2 == pid {print; exit 1}' 2026-02-18T06:43:24.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:24.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:43:24.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:43:24.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.134 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:24.135 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:24.146 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:24.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-18T06:43:24.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:43:24.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:24.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:24.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:24.148 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:24.149 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:24.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:2 2026-02-18T06:43:24.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:24.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.161 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:24.162 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:43:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:43:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:43:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:3 -KILL 2026-02-18T06:43:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:43:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig=-KILL 2026-02-18T06:43:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:43:24.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:3 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:3 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:43:24.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:43:24.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:43:24.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=73524 2026-02-18T06:43:24.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 73524 ']' 2026-02-18T06:43:24.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill -KILL 73524 2026-02-18T06:43:24.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:43:24.187 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:43:24.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:24.188 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73524 '$2 == pid {print; exit 1}' 2026-02-18T06:43:24.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:43:24.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:43:24.204 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=73524 '$2 == pid {print; exit 1}' 2026-02-18T06:43:24.220 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:24.222 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:24.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:24.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-18T06:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:24.234 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:3 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:24.247 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:24.248 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:43:24.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:43:24.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:43:24.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' journal = snapshot ']' 2026-02-18T06:43:24.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ promote_image cluster1 mirror test_force_promote_user --force 2026-02-18T06:43:24.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:24.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:24.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:24.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local force=--force 2026-02-18T06:43:24.259 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote_user --force 2026-02-18T06:43:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:Image promoted to primary 2026-02-18T06:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirrors cluster1 2026-02-18T06:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:34.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:43:34.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:34.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:0 2026-02-18T06:43:34.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:43:34.302 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:0 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=0 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=0 2026-02-18T06:43:34.303 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:34.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:34.304 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:1 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:1 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=1 2026-02-18T06:43:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:34.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:43:34.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:34.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=1 2026-02-18T06:43:34.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:34.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:34.325 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:2 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:2 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:43:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=2 2026-02-18T06:43:34.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:34.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:43:34.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:34.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=2 2026-02-18T06:43:34.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:34.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:34.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:43:34.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:34.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ start_mirror cluster1:3 2026-02-18T06:43:34.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:43:34.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance 2026-02-18T06:43:34.364 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local val=cluster1:3 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster_var_name=cluster 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local instance_var_name=instance 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cluster=cluster1 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ instance=3 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval cluster=cluster1 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ eval instance=3 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:43:34.365 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-18T06:43:34.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote_user 2026-02-18T06:43:34.384 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote_user stopped 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:43:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster1 mirror test_force_promote_user stopped 2026-02-18T06:43:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:43:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:43:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:43:35.386 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:43:35.387 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster1 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster1 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:43:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:35.392 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:35.405 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:35.446 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:35.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:35.452 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:35.460 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:35.461 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:35.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:35.502 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:35.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:35.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:35.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:35.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:35.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:35.516 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:35.517 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:35.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:35.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:35.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:35.561 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:35.563 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:35.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:35.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:35.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:35.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:35.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:35.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:35.613 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:35.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:35.614 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 1 2026-02-18T06:43:36.615 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:36.616 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:36.619 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:36.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:36.628 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:36.629 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:36.683 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:36.684 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:36.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:36.726 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:36.729 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:36.730 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:36.731 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:36.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:36.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:36.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:36.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:36.782 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:36.783 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:36.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:36.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:36.785 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:36.792 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:36.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:36.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:36.793 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:36.833 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:36.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:36.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:36.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:36.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 2 2026-02-18T06:43:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:38.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:38.838 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:38.839 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:38.840 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:38.850 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:38.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:38.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:38.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:38.903 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:38.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:38.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:38.945 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:38.946 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:38.947 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:38.999 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:39.000 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:39.001 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:39.002 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:39.003 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:39.013 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:39.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:39.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:39.054 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:39.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:39.055 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 4 2026-02-18T06:43:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:43.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:43.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:43.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:43.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:43.061 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:43.129 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:43.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:43.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:43.130 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:43.169 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:43.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.173 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:43.174 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:43.175 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:43.224 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:43.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:43.227 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:43.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:43.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:43.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:43.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:43.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:43.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:43.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:43.237 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:43.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:43.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:43.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:43.279 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 8 2026-02-18T06:43:51.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:51.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:51.287 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:51.296 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:51.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:51.340 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:51.341 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:51.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:51.343 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:51.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:51.351 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:51.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:51.352 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:51.395 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:51.396 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:51.409 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:51.453 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:51.454 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:51.455 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:51.463 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:51.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:51.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:51.464 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:51.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:51.506 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 8 2026-02-18T06:43:59.507 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-18T06:43:59.508 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:43:59.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:59.510 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:59.511 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:0 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:43:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:43:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:43:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:43:59.513 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:59.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:59.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:43:59.523 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:59.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:59.565 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:59.569 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:43:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:43:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:43:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:43:59.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:59.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:59.620 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:59.624 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:2 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:43:59.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:59.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:59.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:43:59.634 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-18T06:43:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:rbd mirror status : get status for rbd mirror 2026-02-18T06:43:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:admin_socket: invalid command 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:43:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster1:3 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:43:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster1 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster1 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:43:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:43:59.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:59.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:59.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:43:59.691 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:43:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:43:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:43:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:43:59.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:43:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:43:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:43:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote_user 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state=stopped 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local s 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:43:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:44:00.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-18T06:44:00.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:00.740 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:00.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:00.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local test_state=stopped 2026-02-18T06:44:00.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status_result 2026-02-18T06:44:00.741 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local current_state=stopped 2026-02-18T06:44:00.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:44:00.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_instance=cluster2 2026-02-18T06:44:00.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:44:00.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:00.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance=cluster2 2026-02-18T06:44:00.742 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local loop_instance 2026-02-18T06:44:00.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:44:00.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sleep 0 2026-02-18T06:44:00.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ grep -i state 2026-02-18T06:44:00.743 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-18T06:44:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ seq 0 3 2026-02-18T06:44:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote_user 2026-02-18T06:44:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:44:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ shift 2026-02-18T06:44:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local local_cluster=cluster2:0 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=local_cluster 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval local_cluster=cluster2 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ local_cluster=cluster2 2026-02-18T06:44:00.748 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:44:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:44:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:44:00.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:44:00.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:44:00.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:44:00.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-18T06:44:00.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ return 0 2026-02-18T06:44:00.806 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status_result=' "state": "Stopped"' 2026-02-18T06:44:00.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo ' "state": "Stopped"' 2026-02-18T06:44:00.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep -i Replaying 2026-02-18T06:44:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test stopped = stopped 2026-02-18T06:44:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:44:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote_user up+stopped 2026-02-18T06:44:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:44:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:44:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:44:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:44:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:44:00.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:44:01.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote_user up+stopped '' '' 2026-02-18T06:44:01.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:44:01.810 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote_user 2026-02-18T06:44:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote_user: 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: df6ef9ac-725a-40c3-be08-c153dedb65f0 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.3 on trial130 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:43:59 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster2 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:44:01' 2026-02-18T06:44:01.842 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:44:01.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:44:01.843 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:44:01.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:44:01.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:44:01.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:44:01.844 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:44:01.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:44:01.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:44:01.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.3 on trial130 2026-02-18T06:44:01.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:44:01.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:43:59' 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote_user up+stopped '2026-02-18 06:43:59' 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:43:59' 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:44:01.849 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-18T06:44:01.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK159545959mirror.0trial13020.2.0-670-gadc19233trueOK59615966mirror.1trial13020.2.0-670-gadc19233falseOK59685973mirror.2trial13020.2.0-670-gadc19233falseOK59785981mirror.3trial13020.2.0-670-gadc19233falseOK62266229mirror.0trial13020.2.0-670-gadc19233falseOK62316236mirror.1trial13020.2.0-670-gadc19233falseOK62386242mirror.2trial13020.2.0-670-gadc19233falseOK62446250mirror.3trial13020.2.0-670-gadc19233trueOKtest_force_promote_userdf6ef9ac-725a-40c3-be08-c153dedb65f0up+stoppedlocal image is primary62446250mirror.3trial1302026-02-18 06:43:59cluster2ee4dbd21-9013-44f3-9ef7-aff3b2fab932up+stoppedlocal image is primary2026-02-18 06:44:01' 2026-02-18T06:44:01.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:44:01.881 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-18T06:44:01.882 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:43:59' 2026-02-18T06:44:01.883 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-18T06:44:01.884 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:44:01.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:44:01.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:44:01.885 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:44:01.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:44:01.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:44:01.886 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-18T06:44:01.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 1 2026-02-18T06:44:02.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped '' '' 2026-02-18T06:44:02.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:02.887 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local description_pattern= 2026-02-18T06:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local service_pattern= 2026-02-18T06:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote_user 2026-02-18T06:44:02.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='test_force_promote_user: 2026-02-18T06:44:02.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: global_id: df6ef9ac-725a-40c3-be08-c153dedb65f0 2026-02-18T06:44:02.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:44:02.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:44:02.918 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: service: mirror.0 on trial130 2026-02-18T06:44:02.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:44:01 2026-02-18T06:44:02.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: peer_sites: 2026-02-18T06:44:02.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: name: cluster1 2026-02-18T06:44:02.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: state: up+stopped 2026-02-18T06:44:02.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: description: local image is primary 2026-02-18T06:44:02.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: last_update: 2026-02-18 06:43:59' 2026-02-18T06:44:02.919 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ state: .*up+stopped' 2026-02-18T06:44:02.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: state: up+stopped 2026-02-18T06:44:02.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^ description: .*' 2026-02-18T06:44:02.920 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: description: local image is primary 2026-02-18T06:44:02.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n '' ']' 2026-02-18T06:44:02.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:44:02.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep '^up+' 2026-02-18T06:44:02.921 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:44:02.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep 'service: *mirror..* on ' 2026-02-18T06:44:02.922 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: service: mirror.0 on trial130 2026-02-18T06:44:02.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update 2026-02-18T06:44:02.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:44:01' 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote_user up+stopped '2026-02-18 06:44:01' 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local state_pattern=up+stopped 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local 'prev_last_update=2026-02-18 06:44:01' 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local status 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ CEPH_ARGS= 2026-02-18T06:44:02.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-18T06:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ status='OKOKOK167336738mirror.0trial13020.2.0-670-gadc19233trueOK67406743mirror.1trial13020.2.0-670-gadc19233falseOK67476750mirror.2trial13020.2.0-670-gadc19233falseOK67576763mirror.3trial13020.2.0-670-gadc19233falseOKtest_force_promote_userdf6ef9ac-725a-40c3-be08-c153dedb65f0up+stoppedlocal image is primary67336738mirror.0trial1302026-02-18 06:44:01cluster130a4ae41-2c9e-49d5-8ef7-84071acefee8up+stoppedlocal image is primary2026-02-18 06:43:59' 2026-02-18T06:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local last_update state 2026-02-18T06:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-18T06:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ last_update='2026-02-18 06:44:01' 2026-02-18T06:44:02.957 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-18T06:44:02.958 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ state=up+stopped 2026-02-18T06:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo up+stopped 2026-02-18T06:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ grep up+stopped 2026-02-18T06:44:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:up+stopped 2026-02-18T06:44:02.960 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster1 mirror test_force_promote_user 100 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:44:02.961 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:44:03.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:44:03.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:44:03.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 8333.49 bytes/sec: 33 MiB/s 2026-02-18T06:44:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ write_image cluster2 mirror test_force_promote_user 100 2026-02-18T06:44:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local count=100 2026-02-18T06:44:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local size= 2026-02-18T06:44:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ size=4096 2026-02-18T06:44:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-18T06:44:03.311 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-18T06:44:03.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-18T06:44:03.324 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:elapsed: 0 ops: 100 ops/sec: 7142.99 bytes/sec: 28 MiB/s 2026-02-18T06:44:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster1 mirror test_force_promote_user 2026-02-18T06:44:03.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:44:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:03.332 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster1 mirror test_force_promote_user 2026-02-18T06:44:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:44:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_force_promote_user 2026-02-18T06:44:04.238 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-18T06:44:04.243 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster1 rm mirror/test_force_promote_user 2026-02-18T06:44:04.552 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-18T06:44:04.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:44:04.556 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image_retry cluster2 mirror test_force_promote_user 2026-02-18T06:44:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ remove_image cluster2 mirror test_force_promote_user 2026-02-18T06:44:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pool=mirror 2026-02-18T06:44:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local image=test_force_promote_user 2026-02-18T06:44:04.558 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_force_promote_user 2026-02-18T06:44:06.579 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-18T06:44:06.583 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rbd --cluster=cluster2 rm mirror/test_force_promote_user 2026-02-18T06:44:07.088 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-18T06:44:07.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ return 0 2026-02-18T06:44:07.092 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ cleanup 0 2026-02-18T06:44:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local error_code=0 2026-02-18T06:44:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ set +e 2026-02-18T06:44:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -ne 0 ']' 2026-02-18T06:44:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z '' ']' 2026-02-18T06:44:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster instance 2026-02-18T06:44:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:44:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-18T06:44:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror' does not exist 2026-02-18T06:44:08.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:44:08.305 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-18T06:44:08.639 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror' does not exist 2026-02-18T06:44:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:44:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-18T06:44:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror_parent' does not exist 2026-02-18T06:44:09.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ CEPH_ARGS= 2026-02-18T06:44:09.307 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-18T06:44:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:pool 'mirror_parent' does not exist 2026-02-18T06:44:09.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-18T06:44:09.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster1 2026-02-18T06:44:09.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1 2026-02-18T06:44:09.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:09.655 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:44:09.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:09.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:0 '' 2026-02-18T06:44:09.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:0 2026-02-18T06:44:09.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:09.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:09.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:09.658 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:0 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:0 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:0 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:44:09.659 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:44:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:44:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:44:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:44:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:44:09.671 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:44:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=78579 2026-02-18T06:44:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 78579 ']' 2026-02-18T06:44:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 78579 2026-02-18T06:44:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:09.673 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78579 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:09.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.690 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78579 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-18T06:44:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:44:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.707 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:09.708 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:44:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:44:09.719 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:44:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-18T06:44:09.721 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:0 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:44:09.722 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:44:09.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:09.723 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-18T06:44:09.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:44:09.733 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-18T06:44:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:0 2026-02-18T06:44:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:0 2026-02-18T06:44:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-18T06:44:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:0 2026-02-18T06:44:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:44:09.737 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:09.738 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-18T06:44:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:44:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-18T06:44:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:1 '' 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:1 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:1 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:1 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:1 2026-02-18T06:44:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:44:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:44:09.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:44:09.759 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:44:09.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=78614 2026-02-18T06:44:09.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 78614 ']' 2026-02-18T06:44:09.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 78614 2026-02-18T06:44:09.760 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:09.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:09.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.761 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78614 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78614 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-18T06:44:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:44:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:44:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:44:09.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:44:09.805 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:44:09.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-18T06:44:09.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:1 2026-02-18T06:44:09.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:09.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.807 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:44:09.808 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:09.809 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-18T06:44:09.818 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:44:09.819 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-18T06:44:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:1 2026-02-18T06:44:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:1 2026-02-18T06:44:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.821 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:1 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:44:09.822 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:44:09.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:09.823 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-18T06:44:09.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:44:09.831 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-18T06:44:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:2 '' 2026-02-18T06:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:2 2026-02-18T06:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:2 2026-02-18T06:44:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:2 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:2 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-18T06:44:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:44:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:44:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:44:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:44:09.846 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=78684 2026-02-18T06:44:09.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 78684 ']' 2026-02-18T06:44:09.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 78684 2026-02-18T06:44:09.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:09.847 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:09.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.848 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78684 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78684 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-18T06:44:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:44:09.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:09.880 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:44:09.889 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:44:09.890 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:44:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-18T06:44:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:2 2026-02-18T06:44:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-18T06:44:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-18T06:44:09.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:44:09.904 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-18T06:44:09.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:2 2026-02-18T06:44:09.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:2 2026-02-18T06:44:09.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-18T06:44:09.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:2 2026-02-18T06:44:09.906 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-18T06:44:09.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:44:09.917 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-18T06:44:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster1:3 '' 2026-02-18T06:44:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster1:3 2026-02-18T06:44:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster1:3 2026-02-18T06:44:09.924 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster1:3 2026-02-18T06:44:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:44:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster1:3 2026-02-18T06:44:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:09.925 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:09.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:09.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-18T06:44:09.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster1 2026-02-18T06:44:09.926 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster1 2026-02-18T06:44:09.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:44:09.927 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:44:09.928 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:44:09.931 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:44:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:44:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=78736 2026-02-18T06:44:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 78736 ']' 2026-02-18T06:44:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 78736 2026-02-18T06:44:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78736 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:09.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:09.950 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=78736 '$2 == pid {print; exit 1}' 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:44:09.977 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:44:09.978 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster1:3 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:44:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:44:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster1 2026-02-18T06:44:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster1 2026-02-18T06:44:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:44:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:09.981 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-18T06:44:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:44:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-18T06:44:09.993 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster1:3 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1:3 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster1:3 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster1 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:44:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-18T06:44:09.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster1 2026-02-18T06:44:09.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster1 2026-02-18T06:44:09.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:44:09.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:09.995 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-18T06:44:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:44:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-18T06:44:10.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-18T06:44:10.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirrors cluster2 2026-02-18T06:44:10.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2 2026-02-18T06:44:10.006 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ seq 0 3 2026-02-18T06:44:10.008 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:0 '' 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:0 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:0 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:0 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:44:10.009 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:0 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=0 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=0 2026-02-18T06:44:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-18T06:44:10.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:44:10.020 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:44:10.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=77136 2026-02-18T06:44:10.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 77136 ']' 2026-02-18T06:44:10.021 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 77136 2026-02-18T06:44:10.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:10.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:10.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.022 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77136 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:10.038 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.039 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77136 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster1 2026-02-18T06:44:10.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:44:10.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:10.056 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:44:10.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:44:10.067 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:44:10.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-18T06:44:10.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:0 2026-02-18T06:44:10.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:10.069 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:44:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-18T06:44:10.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:44:10.081 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-18T06:44:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:0 2026-02-18T06:44:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:0 2026-02-18T06:44:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:0 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=0 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=0 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=0 2026-02-18T06:44:10.084 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-18T06:44:10.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:44:10.095 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-18T06:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:1 '' 2026-02-18T06:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:1 2026-02-18T06:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:10.096 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:10.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:1 2026-02-18T06:44:10.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:1 2026-02-18T06:44:10.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:10.098 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:1 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=1 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=1 2026-02-18T06:44:10.099 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-18T06:44:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:44:10.110 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:44:10.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=77180 2026-02-18T06:44:10.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 77180 ']' 2026-02-18T06:44:10.111 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 77180 2026-02-18T06:44:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.112 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77180 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.126 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:10.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.127 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77180 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.142 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster1 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:44:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:44:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:10.144 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:44:10.153 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:44:10.154 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:44:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-18T06:44:10.155 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:1 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:44:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-18T06:44:10.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:44:10.168 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-18T06:44:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:1 2026-02-18T06:44:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:1 2026-02-18T06:44:10.170 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:1 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=1 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.171 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=1 2026-02-18T06:44:10.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=1 2026-02-18T06:44:10.172 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-18T06:44:10.181 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:44:10.182 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-18T06:44:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:2 '' 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:2 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:2 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:2 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:2 2026-02-18T06:44:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=2 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=2 2026-02-18T06:44:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-18T06:44:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:44:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:44:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=77241 2026-02-18T06:44:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 77241 ']' 2026-02-18T06:44:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 77241 2026-02-18T06:44:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77241 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.214 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:10.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.215 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77241 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster1 2026-02-18T06:44:10.230 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:44:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:10.232 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:44:10.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:44:10.242 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:2 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.244 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:44:10.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:44:10.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:44:10.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:10.245 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-18T06:44:10.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:44:10.255 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:2 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:2 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:2 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=2 2026-02-18T06:44:10.257 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-18T06:44:10.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:44:10.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=2 2026-02-18T06:44:10.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=2 2026-02-18T06:44:10.258 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-18T06:44:10.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:44:10.268 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ stop_mirror cluster2:3 '' 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local cluster=cluster2:3 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local sig= 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test -n '' 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ local pid 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ daemon_pid_file cluster2:3 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster=cluster2:3 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance 2026-02-18T06:44:10.271 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local val=cluster2:3 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local cluster_var_name=cluster 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local instance_var_name=instance 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval cluster=cluster2 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ cluster=cluster2 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ eval instance=3 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ instance=3 2026-02-18T06:44:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-18T06:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:44:10.283 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ pid=77309 2026-02-18T06:44:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -n 77309 ']' 2026-02-18T06:44:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ kill 77309 2026-02-18T06:44:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-18T06:44:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ sleep 0 2026-02-18T06:44:10.284 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.285 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77309 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ break 2026-02-18T06:44:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ ps auxww 2026-02-18T06:44:10.300 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ awk -v pid=77309 '$2 == pid {print; exit 1}' 2026-02-18T06:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster1 2026-02-18T06:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster1 2026-02-18T06:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:10.317 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:44:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:44:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:44:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-18T06:44:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local local_cluster=cluster2:3 2026-02-18T06:44:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2 2026-02-18T06:44:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-18T06:44:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:44:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=local_cluster 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval local_cluster=cluster2 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ local_cluster=cluster2 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-18T06:44:10.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:44:10.342 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-18T06:44:10.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ daemon_pid_file cluster2:3 2026-02-18T06:44:10.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster=cluster2:3 2026-02-18T06:44:10.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance 2026-02-18T06:44:10.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-18T06:44:10.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local val=cluster2:3 2026-02-18T06:44:10.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local cluster_var_name=cluster 2026-02-18T06:44:10.344 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ local instance_var_name=instance 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ cluster=cluster2 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ instance=3 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval cluster=cluster2 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ cluster=cluster2 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ eval instance=3 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ instance=3 2026-02-18T06:44:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-18T06:44:10.355 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:44:10.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-18T06:44:10.356 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' -z 1 ']' 2026-02-18T06:44:10.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ test '' = /tmp/tmp.mzUeR8FHMy 2026-02-18T06:44:10.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ rm -Rf /tmp/tmp.mzUeR8FHMy 2026-02-18T06:44:10.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ '[' 0 -eq 0 ']' 2026-02-18T06:44:10.357 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ echo OK 2026-02-18T06:44:10.358 INFO:tasks.workunit.cluster1.client.mirror.trial130.stdout:OK 2026-02-18T06:44:10.359 INFO:tasks.workunit.cluster1.client.mirror.trial130.stderr:+ exit 0 2026-02-18T06:44:10.359 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-18T06:44:10.359 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2026-02-18T06:44:10.427 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror.sh'] on cluster1.client.mirror... 2026-02-18T06:44:10.427 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2026-02-18T06:44:10.779 DEBUG:teuthology.parallel:result is None 2026-02-18T06:44:10.779 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-18T06:44:10.804 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-18T06:44:10.805 DEBUG:teuthology.orchestra.run.trial130:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-18T06:44:10.861 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-18T06:44:10.861 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-18T06:44:10.878 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-18T06:44:10.878 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-18T06:44:11.080 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:44:11.080 INFO:teuthology.orchestra.run.trial130.stderr:dumped all 2026-02-18T06:44:11.089 INFO:teuthology.orchestra.run.trial130.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2026-02-18T06:44:09.577083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15583228,"kb_used_data":14159852,"kb_used_omap":1952,"kb_used_meta":782367,"kb_avail":267532292,"statfs":{"total":289910292480,"available":273953067008,"internally_reserved":0,"allocated":14499688448,"data_stored":14497411072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1998975,"internal_metadata":801144705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14973660576,"num_objects":-2330,"num_object_clones":-115,"num_object_copies":-4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2330,"num_whiteouts":0,"num_read":-88147,"num_read_kb":-517580,"num_write":-22408,"num_write_kb":-21061282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-1,"num_objects_omap":-30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.237840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":325,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.574913+0000","last_change":"2026-02-18T06:20:33.279922+0000","last_active":"2026-02-18T06:44:04.574913+0000","last_peered":"2026-02-18T06:44:04.574913+0000","last_clean":"2026-02-18T06:44:04.574913+0000","last_became_active":"2026-02-18T06:20:31.279056+0000","last_became_peered":"2026-02-18T06:20:31.279056+0000","last_unstale":"2026-02-18T06:44:04.574913+0000","last_undegraded":"2026-02-18T06:44:04.574913+0000","last_fullsized":"2026-02-18T06:44:04.574913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:34:10.245401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053659699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":325,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.575094+0000","last_change":"2026-02-18T06:20:33.279639+0000","last_active":"2026-02-18T06:44:04.575094+0000","last_peered":"2026-02-18T06:44:04.575094+0000","last_clean":"2026-02-18T06:44:04.575094+0000","last_became_active":"2026-02-18T06:20:31.279113+0000","last_became_peered":"2026-02-18T06:20:31.279113+0000","last_unstale":"2026-02-18T06:44:04.575094+0000","last_undegraded":"2026-02-18T06:44:04.575094+0000","last_fullsized":"2026-02-18T06:44:04.575094+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:36:05.161609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030664000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":326,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.573989+0000","last_change":"2026-02-18T06:20:33.279798+0000","last_active":"2026-02-18T06:44:04.573989+0000","last_peered":"2026-02-18T06:44:04.573989+0000","last_clean":"2026-02-18T06:44:04.573989+0000","last_became_active":"2026-02-18T06:20:31.279035+0000","last_became_peered":"2026-02-18T06:20:31.279035+0000","last_unstale":"2026-02-18T06:44:04.573989+0000","last_undegraded":"2026-02-18T06:44:04.573989+0000","last_fullsized":"2026-02-18T06:44:04.573989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:28:19.490146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043416700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":326,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.573999+0000","last_change":"2026-02-18T06:20:33.279641+0000","last_active":"2026-02-18T06:44:04.573999+0000","last_peered":"2026-02-18T06:44:04.573999+0000","last_clean":"2026-02-18T06:44:04.573999+0000","last_became_active":"2026-02-18T06:20:31.278965+0000","last_became_peered":"2026-02-18T06:20:31.278965+0000","last_unstale":"2026-02-18T06:44:04.573999+0000","last_undegraded":"2026-02-18T06:44:04.573999+0000","last_fullsized":"2026-02-18T06:44:04.573999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:25:34.395735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027168500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":329,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.570547+0000","last_change":"2026-02-18T06:20:33.278364+0000","last_active":"2026-02-18T06:44:04.570547+0000","last_peered":"2026-02-18T06:44:04.570547+0000","last_clean":"2026-02-18T06:44:04.570547+0000","last_became_active":"2026-02-18T06:20:31.279064+0000","last_became_peered":"2026-02-18T06:20:31.279064+0000","last_unstale":"2026-02-18T06:44:04.570547+0000","last_undegraded":"2026-02-18T06:44:04.570547+0000","last_fullsized":"2026-02-18T06:44:04.570547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:11:20.458617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020610299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":164,"state":"active+clean","last_fresh":"2026-02-18T06:44:05.573077+0000","last_change":"2026-02-18T06:20:33.748366+0000","last_active":"2026-02-18T06:44:05.573077+0000","last_peered":"2026-02-18T06:44:05.573077+0000","last_clean":"2026-02-18T06:44:05.573077+0000","last_became_active":"2026-02-18T06:20:31.279036+0000","last_became_peered":"2026-02-18T06:20:31.279036+0000","last_unstale":"2026-02-18T06:44:05.573077+0000","last_undegraded":"2026-02-18T06:44:05.573077+0000","last_fullsized":"2026-02-18T06:44:05.573077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:52:51.049171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017900300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":164,"state":"active+clean","last_fresh":"2026-02-18T06:44:05.571755+0000","last_change":"2026-02-18T06:20:33.748330+0000","last_active":"2026-02-18T06:44:05.571755+0000","last_peered":"2026-02-18T06:44:05.571755+0000","last_clean":"2026-02-18T06:44:05.571755+0000","last_became_active":"2026-02-18T06:20:31.279067+0000","last_became_peered":"2026-02-18T06:20:31.279067+0000","last_unstale":"2026-02-18T06:44:05.571755+0000","last_undegraded":"2026-02-18T06:44:05.571755+0000","last_fullsized":"2026-02-18T06:44:05.571755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:55:16.535994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017421299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":327,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.574545+0000","last_change":"2026-02-18T06:20:33.279767+0000","last_active":"2026-02-18T06:44:04.574545+0000","last_peered":"2026-02-18T06:44:04.574545+0000","last_clean":"2026-02-18T06:44:04.574545+0000","last_became_active":"2026-02-18T06:20:31.278647+0000","last_became_peered":"2026-02-18T06:20:31.278647+0000","last_unstale":"2026-02-18T06:44:04.574545+0000","last_undegraded":"2026-02-18T06:44:04.574545+0000","last_fullsized":"2026-02-18T06:44:04.574545+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:33:35.594020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040074499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":417,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.575467+0000","last_change":"2026-02-18T06:20:28.711581+0000","last_active":"2026-02-18T06:44:04.575467+0000","last_peered":"2026-02-18T06:44:04.575467+0000","last_clean":"2026-02-18T06:44:04.575467+0000","last_became_active":"2026-02-18T06:20:28.710726+0000","last_became_peered":"2026-02-18T06:20:28.710726+0000","last_unstale":"2026-02-18T06:44:04.575467+0000","last_undegraded":"2026-02-18T06:44:04.575467+0000","last_fullsized":"2026-02-18T06:44:04.575467+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_clean_scrub_stamp":"2026-02-18T06:20:27.265730+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T09:48:59.040538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738656,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4839436,"kb_used_data":4071708,"kb_used_omap":792,"kb_used_meta":242599,"kb_avail":89532404,"statfs":{"total":96636764160,"available":91681181696,"internally_reserved":0,"allocated":4169428992,"data_stored":4168717184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":811538,"internal_metadata":248421870},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738655,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4535148,"kb_used_data":4291740,"kb_used_omap":472,"kb_used_meta":242919,"kb_avail":89836692,"statfs":{"total":96636764160,"available":91992772608,"internally_reserved":0,"allocated":4394741760,"data_stored":4394044119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":484234,"internal_metadata":248749174},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738655,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6208644,"kb_used_data":5796404,"kb_used_omap":686,"kb_used_meta":296849,"kb_avail":88163196,"statfs":{"total":96636764160,"available":90279112704,"internally_reserved":0,"allocated":5935517696,"data_stored":5934649769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":703203,"internal_metadata":303973661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:11.089 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-18T06:44:11.270 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:44:11.270 INFO:teuthology.orchestra.run.trial130.stderr:dumped all 2026-02-18T06:44:11.279 INFO:teuthology.orchestra.run.trial130.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2026-02-18T06:44:09.577083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15583228,"kb_used_data":14159852,"kb_used_omap":1952,"kb_used_meta":782367,"kb_avail":267532292,"statfs":{"total":289910292480,"available":273953067008,"internally_reserved":0,"allocated":14499688448,"data_stored":14497411072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1998975,"internal_metadata":801144705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14973660576,"num_objects":-2330,"num_object_clones":-115,"num_object_copies":-4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2330,"num_whiteouts":0,"num_read":-88147,"num_read_kb":-517580,"num_write":-22408,"num_write_kb":-21061282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-1,"num_objects_omap":-30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.237840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":325,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.574913+0000","last_change":"2026-02-18T06:20:33.279922+0000","last_active":"2026-02-18T06:44:04.574913+0000","last_peered":"2026-02-18T06:44:04.574913+0000","last_clean":"2026-02-18T06:44:04.574913+0000","last_became_active":"2026-02-18T06:20:31.279056+0000","last_became_peered":"2026-02-18T06:20:31.279056+0000","last_unstale":"2026-02-18T06:44:04.574913+0000","last_undegraded":"2026-02-18T06:44:04.574913+0000","last_fullsized":"2026-02-18T06:44:04.574913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:34:10.245401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053659699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":325,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.575094+0000","last_change":"2026-02-18T06:20:33.279639+0000","last_active":"2026-02-18T06:44:04.575094+0000","last_peered":"2026-02-18T06:44:04.575094+0000","last_clean":"2026-02-18T06:44:04.575094+0000","last_became_active":"2026-02-18T06:20:31.279113+0000","last_became_peered":"2026-02-18T06:20:31.279113+0000","last_unstale":"2026-02-18T06:44:04.575094+0000","last_undegraded":"2026-02-18T06:44:04.575094+0000","last_fullsized":"2026-02-18T06:44:04.575094+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:36:05.161609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030664000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":326,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.573989+0000","last_change":"2026-02-18T06:20:33.279798+0000","last_active":"2026-02-18T06:44:04.573989+0000","last_peered":"2026-02-18T06:44:04.573989+0000","last_clean":"2026-02-18T06:44:04.573989+0000","last_became_active":"2026-02-18T06:20:31.279035+0000","last_became_peered":"2026-02-18T06:20:31.279035+0000","last_unstale":"2026-02-18T06:44:04.573989+0000","last_undegraded":"2026-02-18T06:44:04.573989+0000","last_fullsized":"2026-02-18T06:44:04.573989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:28:19.490146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043416700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":326,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.573999+0000","last_change":"2026-02-18T06:20:33.279641+0000","last_active":"2026-02-18T06:44:04.573999+0000","last_peered":"2026-02-18T06:44:04.573999+0000","last_clean":"2026-02-18T06:44:04.573999+0000","last_became_active":"2026-02-18T06:20:31.278965+0000","last_became_peered":"2026-02-18T06:20:31.278965+0000","last_unstale":"2026-02-18T06:44:04.573999+0000","last_undegraded":"2026-02-18T06:44:04.573999+0000","last_fullsized":"2026-02-18T06:44:04.573999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:25:34.395735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027168500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":329,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.570547+0000","last_change":"2026-02-18T06:20:33.278364+0000","last_active":"2026-02-18T06:44:04.570547+0000","last_peered":"2026-02-18T06:44:04.570547+0000","last_clean":"2026-02-18T06:44:04.570547+0000","last_became_active":"2026-02-18T06:20:31.279064+0000","last_became_peered":"2026-02-18T06:20:31.279064+0000","last_unstale":"2026-02-18T06:44:04.570547+0000","last_undegraded":"2026-02-18T06:44:04.570547+0000","last_fullsized":"2026-02-18T06:44:04.570547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:11:20.458617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020610299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":164,"state":"active+clean","last_fresh":"2026-02-18T06:44:05.573077+0000","last_change":"2026-02-18T06:20:33.748366+0000","last_active":"2026-02-18T06:44:05.573077+0000","last_peered":"2026-02-18T06:44:05.573077+0000","last_clean":"2026-02-18T06:44:05.573077+0000","last_became_active":"2026-02-18T06:20:31.279036+0000","last_became_peered":"2026-02-18T06:20:31.279036+0000","last_unstale":"2026-02-18T06:44:05.573077+0000","last_undegraded":"2026-02-18T06:44:05.573077+0000","last_fullsized":"2026-02-18T06:44:05.573077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:52:51.049171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017900300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":164,"state":"active+clean","last_fresh":"2026-02-18T06:44:05.571755+0000","last_change":"2026-02-18T06:20:33.748330+0000","last_active":"2026-02-18T06:44:05.571755+0000","last_peered":"2026-02-18T06:44:05.571755+0000","last_clean":"2026-02-18T06:44:05.571755+0000","last_became_active":"2026-02-18T06:20:31.279067+0000","last_became_peered":"2026-02-18T06:20:31.279067+0000","last_unstale":"2026-02-18T06:44:05.571755+0000","last_undegraded":"2026-02-18T06:44:05.571755+0000","last_fullsized":"2026-02-18T06:44:05.571755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:55:16.535994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017421299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":327,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.574545+0000","last_change":"2026-02-18T06:20:33.279767+0000","last_active":"2026-02-18T06:44:04.574545+0000","last_peered":"2026-02-18T06:44:04.574545+0000","last_clean":"2026-02-18T06:44:04.574545+0000","last_became_active":"2026-02-18T06:20:31.278647+0000","last_became_peered":"2026-02-18T06:20:31.278647+0000","last_unstale":"2026-02-18T06:44:04.574545+0000","last_undegraded":"2026-02-18T06:44:04.574545+0000","last_fullsized":"2026-02-18T06:44:04.574545+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:33:35.594020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040074499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":417,"reported_epoch":163,"state":"active+clean","last_fresh":"2026-02-18T06:44:04.575467+0000","last_change":"2026-02-18T06:20:28.711581+0000","last_active":"2026-02-18T06:44:04.575467+0000","last_peered":"2026-02-18T06:44:04.575467+0000","last_clean":"2026-02-18T06:44:04.575467+0000","last_became_active":"2026-02-18T06:20:28.710726+0000","last_became_peered":"2026-02-18T06:20:28.710726+0000","last_unstale":"2026-02-18T06:44:04.575467+0000","last_undegraded":"2026-02-18T06:44:04.575467+0000","last_fullsized":"2026-02-18T06:44:04.575467+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_clean_scrub_stamp":"2026-02-18T06:20:27.265730+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T09:48:59.040538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738656,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4839436,"kb_used_data":4071708,"kb_used_omap":792,"kb_used_meta":242599,"kb_avail":89532404,"statfs":{"total":96636764160,"available":91681181696,"internally_reserved":0,"allocated":4169428992,"data_stored":4168717184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":811538,"internal_metadata":248421870},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738655,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4535148,"kb_used_data":4291740,"kb_used_omap":472,"kb_used_meta":242919,"kb_avail":89836692,"statfs":{"total":96636764160,"available":91992772608,"internally_reserved":0,"allocated":4394741760,"data_stored":4394044119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":484234,"internal_metadata":248749174},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738655,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6208644,"kb_used_data":5796404,"kb_used_omap":686,"kb_used_meta":296849,"kb_avail":88163196,"statfs":{"total":96636764160,"available":90279112704,"internally_reserved":0,"allocated":5935517696,"data_stored":5934649769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":703203,"internal_metadata":303973661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:11.280 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-18T06:44:11.280 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-18T06:44:11.461 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:44:11.462 INFO:teuthology.orchestra.run.trial130.stderr:dumped all 2026-02-18T06:44:11.470 INFO:teuthology.orchestra.run.trial130.stdout:{"pg_ready":true,"pg_map":{"version":878,"stamp":"2026-02-18T06:44:11.340096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":196,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":825808,"kb_used_data":5344,"kb_used_omap":2008,"kb_used_meta":818407,"kb_avail":282289712,"statfs":{"total":289910292480,"available":289064665088,"internally_reserved":0,"allocated":5472256,"data_stored":3323659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2056887,"internal_metadata":838049097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10789283325,"num_objects":-2091,"num_object_clones":-115,"num_object_copies":-4182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2091,"num_whiteouts":0,"num_read":-89041,"num_read_kb":-570099,"num_write":-22884,"num_write_kb":-21061840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-1,"num_objects_omap":-30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.772948"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.583579+0000","last_change":"2026-02-18T06:20:33.279922+0000","last_active":"2026-02-18T06:44:09.583579+0000","last_peered":"2026-02-18T06:44:09.583579+0000","last_clean":"2026-02-18T06:44:09.583579+0000","last_became_active":"2026-02-18T06:20:31.279056+0000","last_became_peered":"2026-02-18T06:20:31.279056+0000","last_unstale":"2026-02-18T06:44:09.583579+0000","last_undegraded":"2026-02-18T06:44:09.583579+0000","last_fullsized":"2026-02-18T06:44:09.583579+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:34:10.245401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053659699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.583871+0000","last_change":"2026-02-18T06:20:33.279639+0000","last_active":"2026-02-18T06:44:09.583871+0000","last_peered":"2026-02-18T06:44:09.583871+0000","last_clean":"2026-02-18T06:44:09.583871+0000","last_became_active":"2026-02-18T06:20:31.279113+0000","last_became_peered":"2026-02-18T06:20:31.279113+0000","last_unstale":"2026-02-18T06:44:09.583871+0000","last_undegraded":"2026-02-18T06:44:09.583871+0000","last_fullsized":"2026-02-18T06:44:09.583871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:36:05.161609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030664000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":334,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.582198+0000","last_change":"2026-02-18T06:20:33.279798+0000","last_active":"2026-02-18T06:44:09.582198+0000","last_peered":"2026-02-18T06:44:09.582198+0000","last_clean":"2026-02-18T06:44:09.582198+0000","last_became_active":"2026-02-18T06:20:31.279035+0000","last_became_peered":"2026-02-18T06:20:31.279035+0000","last_unstale":"2026-02-18T06:44:09.582198+0000","last_undegraded":"2026-02-18T06:44:09.582198+0000","last_fullsized":"2026-02-18T06:44:09.582198+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:28:19.490146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043416700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":334,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.583169+0000","last_change":"2026-02-18T06:20:33.279641+0000","last_active":"2026-02-18T06:44:09.583169+0000","last_peered":"2026-02-18T06:44:09.583169+0000","last_clean":"2026-02-18T06:44:09.583169+0000","last_became_active":"2026-02-18T06:20:31.278965+0000","last_became_peered":"2026-02-18T06:20:31.278965+0000","last_unstale":"2026-02-18T06:44:09.583169+0000","last_undegraded":"2026-02-18T06:44:09.583169+0000","last_fullsized":"2026-02-18T06:44:09.583169+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:25:34.395735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027168500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":337,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.581401+0000","last_change":"2026-02-18T06:20:33.278364+0000","last_active":"2026-02-18T06:44:09.581401+0000","last_peered":"2026-02-18T06:44:09.581401+0000","last_clean":"2026-02-18T06:44:09.581401+0000","last_became_active":"2026-02-18T06:20:31.279064+0000","last_became_peered":"2026-02-18T06:20:31.279064+0000","last_unstale":"2026-02-18T06:44:09.581401+0000","last_undegraded":"2026-02-18T06:44:09.581401+0000","last_fullsized":"2026-02-18T06:44:09.581401+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:11:20.458617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020610299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.581462+0000","last_change":"2026-02-18T06:20:33.748366+0000","last_active":"2026-02-18T06:44:09.581462+0000","last_peered":"2026-02-18T06:44:09.581462+0000","last_clean":"2026-02-18T06:44:09.581462+0000","last_became_active":"2026-02-18T06:20:31.279036+0000","last_became_peered":"2026-02-18T06:20:31.279036+0000","last_unstale":"2026-02-18T06:44:09.581462+0000","last_undegraded":"2026-02-18T06:44:09.581462+0000","last_fullsized":"2026-02-18T06:44:09.581462+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:52:51.049171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017900300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.580819+0000","last_change":"2026-02-18T06:20:33.748330+0000","last_active":"2026-02-18T06:44:09.580819+0000","last_peered":"2026-02-18T06:44:09.580819+0000","last_clean":"2026-02-18T06:44:09.580819+0000","last_became_active":"2026-02-18T06:20:31.279067+0000","last_became_peered":"2026-02-18T06:20:31.279067+0000","last_unstale":"2026-02-18T06:44:09.580819+0000","last_undegraded":"2026-02-18T06:44:09.580819+0000","last_fullsized":"2026-02-18T06:44:09.580819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:55:16.535994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017421299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":335,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.584730+0000","last_change":"2026-02-18T06:20:33.279767+0000","last_active":"2026-02-18T06:44:09.584730+0000","last_peered":"2026-02-18T06:44:09.584730+0000","last_clean":"2026-02-18T06:44:09.584730+0000","last_became_active":"2026-02-18T06:20:31.278647+0000","last_became_peered":"2026-02-18T06:20:31.278647+0000","last_unstale":"2026-02-18T06:44:09.584730+0000","last_undegraded":"2026-02-18T06:44:09.584730+0000","last_fullsized":"2026-02-18T06:44:09.584730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:33:35.594020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040074499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":425,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.584866+0000","last_change":"2026-02-18T06:20:28.711581+0000","last_active":"2026-02-18T06:44:09.584866+0000","last_peered":"2026-02-18T06:44:09.584866+0000","last_clean":"2026-02-18T06:44:09.584866+0000","last_became_active":"2026-02-18T06:20:28.710726+0000","last_became_peered":"2026-02-18T06:20:28.710726+0000","last_unstale":"2026-02-18T06:44:09.584866+0000","last_undegraded":"2026-02-18T06:44:09.584866+0000","last_fullsized":"2026-02-18T06:44:09.584866+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_clean_scrub_stamp":"2026-02-18T06:20:27.265730+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T09:48:59.040538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738657,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":262848,"kb_used_data":1392,"kb_used_omap":799,"kb_used_meta":260640,"kb_avail":94108992,"statfs":{"total":96636764160,"available":96367607808,"internally_reserved":0,"allocated":1425408,"data_stored":714295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":818782,"internal_metadata":266895778},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738656,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263432,"kb_used_data":1976,"kb_used_omap":494,"kb_used_meta":260945,"kb_avail":94108408,"statfs":{"total":96636764160,"available":96367009792,"internally_reserved":0,"allocated":2023424,"data_stored":1304682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":505992,"internal_metadata":267208568},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738656,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299528,"kb_used_data":1976,"kb_used_omap":714,"kb_used_meta":296821,"kb_avail":94072312,"statfs":{"total":96636764160,"available":96330047488,"internally_reserved":0,"allocated":2023424,"data_stored":1304682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":732113,"internal_metadata":303944751},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:11.472 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-18T06:44:11.655 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:44:11.656 INFO:teuthology.orchestra.run.trial130.stdout:{"epoch":167,"fsid":"59a365c0-617d-43e5-831e-67d120cf5f4b","created":"2026-02-18T06:20:21.254675+0000","modified":"2026-02-18T06:44:09.577511+0000","last_up_change":"2026-02-18T06:20:25.261461+0000","last_in_change":"2026-02-18T06:20:21.927709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:27.157632+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-18T06:20:29.327794+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ae0369e-50e2-440b-8d51-bce7a9a0c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6800","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6801","nonce":1721000025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6802","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6803","nonce":1721000025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6806","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6807","nonce":1721000025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6804","nonce":1721000025},{"type":"v1","addr":"10.20.193.130:6805","nonce":1721000025}]},"public_addr":"10.20.193.130:6801/1721000025","cluster_addr":"10.20.193.130:6803/1721000025","heartbeat_back_addr":"10.20.193.130:6807/1721000025","heartbeat_front_addr":"10.20.193.130:6805/1721000025","state":["exists","up"]},{"osd":1,"uuid":"43da5d80-4629-409f-8241-c6f9b94c3e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6808","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6809","nonce":370005737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6810","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6811","nonce":370005737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6814","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6815","nonce":370005737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6812","nonce":370005737},{"type":"v1","addr":"10.20.193.130:6813","nonce":370005737}]},"public_addr":"10.20.193.130:6809/370005737","cluster_addr":"10.20.193.130:6811/370005737","heartbeat_back_addr":"10.20.193.130:6815/370005737","heartbeat_front_addr":"10.20.193.130:6813/370005737","state":["exists","up"]},{"osd":2,"uuid":"bd78b66b-a532-40e3-a671-f903661484b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6816","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6817","nonce":1080821858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6818","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6819","nonce":1080821858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6822","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6823","nonce":1080821858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.130:6820","nonce":1080821858},{"type":"v1","addr":"10.20.193.130:6821","nonce":1080821858}]},"public_addr":"10.20.193.130:6817/1080821858","cluster_addr":"10.20.193.130:6819/1080821858","heartbeat_back_addr":"10.20.193.130:6823/1080821858","heartbeat_front_addr":"10.20.193.130:6821/1080821858","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-18T06:20:23.696007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-18T06:20:23.765914+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-18T06:20:23.789189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":45,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:44:12.666 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2026-02-18T06:44:12.667 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-18T06:44:12.751 INFO:teuthology.orchestra.run.trial130.stdout:{ 2026-02-18T06:44:12.751 INFO:teuthology.orchestra.run.trial130.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-18T06:44:12.751 INFO:teuthology.orchestra.run.trial130.stdout:} 2026-02-18T06:44:12.757 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2026-02-18T06:44:12.938 INFO:teuthology.orchestra.run.trial130.stderr:instructed osd(s) 0 to deep-scrub 2026-02-18T06:44:12.947 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2026-02-18T06:44:12.948 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-18T06:44:13.030 INFO:teuthology.orchestra.run.trial130.stdout:{ 2026-02-18T06:44:13.030 INFO:teuthology.orchestra.run.trial130.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-18T06:44:13.030 INFO:teuthology.orchestra.run.trial130.stdout:} 2026-02-18T06:44:13.037 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2026-02-18T06:44:13.217 INFO:teuthology.orchestra.run.trial130.stderr:instructed osd(s) 1 to deep-scrub 2026-02-18T06:44:13.227 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2026-02-18T06:44:13.228 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-18T06:44:13.309 INFO:teuthology.orchestra.run.trial130.stdout:{ 2026-02-18T06:44:13.310 INFO:teuthology.orchestra.run.trial130.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-18T06:44:13.310 INFO:teuthology.orchestra.run.trial130.stdout:} 2026-02-18T06:44:13.316 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2026-02-18T06:44:13.497 INFO:teuthology.orchestra.run.trial130.stderr:instructed osd(s) 2 to deep-scrub 2026-02-18T06:44:13.507 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-18T06:44:13.690 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:44:13.690 INFO:teuthology.orchestra.run.trial130.stderr:dumped all 2026-02-18T06:44:13.699 INFO:teuthology.orchestra.run.trial130.stdout:{"pg_ready":true,"pg_map":{"version":879,"stamp":"2026-02-18T06:44:13.340332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":196,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":825808,"kb_used_data":5344,"kb_used_omap":2008,"kb_used_meta":818407,"kb_avail":282289712,"statfs":{"total":289910292480,"available":289064665088,"internally_reserved":0,"allocated":5472256,"data_stored":3323659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2056887,"internal_metadata":838049097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7168621895,"num_objects":-1852,"num_object_clones":-78,"num_object_copies":-3704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1852,"num_whiteouts":0,"num_read":-89436,"num_read_kb":-622090,"num_write":-23156,"num_write_kb":-21062169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-1,"num_objects_omap":-30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.771172"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.583579+0000","last_change":"2026-02-18T06:20:33.279922+0000","last_active":"2026-02-18T06:44:09.583579+0000","last_peered":"2026-02-18T06:44:09.583579+0000","last_clean":"2026-02-18T06:44:09.583579+0000","last_became_active":"2026-02-18T06:20:31.279056+0000","last_became_peered":"2026-02-18T06:20:31.279056+0000","last_unstale":"2026-02-18T06:44:09.583579+0000","last_undegraded":"2026-02-18T06:44:09.583579+0000","last_fullsized":"2026-02-18T06:44:09.583579+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:34:10.245401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053659699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.583871+0000","last_change":"2026-02-18T06:20:33.279639+0000","last_active":"2026-02-18T06:44:09.583871+0000","last_peered":"2026-02-18T06:44:09.583871+0000","last_clean":"2026-02-18T06:44:09.583871+0000","last_became_active":"2026-02-18T06:20:31.279113+0000","last_became_peered":"2026-02-18T06:20:31.279113+0000","last_unstale":"2026-02-18T06:44:09.583871+0000","last_undegraded":"2026-02-18T06:44:09.583871+0000","last_fullsized":"2026-02-18T06:44:09.583871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:36:05.161609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030664000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":334,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.582198+0000","last_change":"2026-02-18T06:20:33.279798+0000","last_active":"2026-02-18T06:44:09.582198+0000","last_peered":"2026-02-18T06:44:09.582198+0000","last_clean":"2026-02-18T06:44:09.582198+0000","last_became_active":"2026-02-18T06:20:31.279035+0000","last_became_peered":"2026-02-18T06:20:31.279035+0000","last_unstale":"2026-02-18T06:44:09.582198+0000","last_undegraded":"2026-02-18T06:44:09.582198+0000","last_fullsized":"2026-02-18T06:44:09.582198+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:28:19.490146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043416700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":334,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.583169+0000","last_change":"2026-02-18T06:20:33.279641+0000","last_active":"2026-02-18T06:44:09.583169+0000","last_peered":"2026-02-18T06:44:09.583169+0000","last_clean":"2026-02-18T06:44:09.583169+0000","last_became_active":"2026-02-18T06:20:31.278965+0000","last_became_peered":"2026-02-18T06:20:31.278965+0000","last_unstale":"2026-02-18T06:44:09.583169+0000","last_undegraded":"2026-02-18T06:44:09.583169+0000","last_fullsized":"2026-02-18T06:44:09.583169+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:25:34.395735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027168500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":337,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.581401+0000","last_change":"2026-02-18T06:20:33.278364+0000","last_active":"2026-02-18T06:44:09.581401+0000","last_peered":"2026-02-18T06:44:09.581401+0000","last_clean":"2026-02-18T06:44:09.581401+0000","last_became_active":"2026-02-18T06:20:31.279064+0000","last_became_peered":"2026-02-18T06:20:31.279064+0000","last_unstale":"2026-02-18T06:44:09.581401+0000","last_undegraded":"2026-02-18T06:44:09.581401+0000","last_fullsized":"2026-02-18T06:44:09.581401+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:11:20.458617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020610299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.581462+0000","last_change":"2026-02-18T06:20:33.748366+0000","last_active":"2026-02-18T06:44:09.581462+0000","last_peered":"2026-02-18T06:44:09.581462+0000","last_clean":"2026-02-18T06:44:09.581462+0000","last_became_active":"2026-02-18T06:20:31.279036+0000","last_became_peered":"2026-02-18T06:20:31.279036+0000","last_unstale":"2026-02-18T06:44:09.581462+0000","last_undegraded":"2026-02-18T06:44:09.581462+0000","last_fullsized":"2026-02-18T06:44:09.581462+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:52:51.049171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017900300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.580819+0000","last_change":"2026-02-18T06:20:33.748330+0000","last_active":"2026-02-18T06:44:09.580819+0000","last_peered":"2026-02-18T06:44:09.580819+0000","last_clean":"2026-02-18T06:44:09.580819+0000","last_became_active":"2026-02-18T06:20:31.279067+0000","last_became_peered":"2026-02-18T06:20:31.279067+0000","last_unstale":"2026-02-18T06:44:09.580819+0000","last_undegraded":"2026-02-18T06:44:09.580819+0000","last_fullsized":"2026-02-18T06:44:09.580819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:55:16.535994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017421299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":335,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.584730+0000","last_change":"2026-02-18T06:20:33.279767+0000","last_active":"2026-02-18T06:44:09.584730+0000","last_peered":"2026-02-18T06:44:09.584730+0000","last_clean":"2026-02-18T06:44:09.584730+0000","last_became_active":"2026-02-18T06:20:31.278647+0000","last_became_peered":"2026-02-18T06:20:31.278647+0000","last_unstale":"2026-02-18T06:44:09.584730+0000","last_undegraded":"2026-02-18T06:44:09.584730+0000","last_fullsized":"2026-02-18T06:44:09.584730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:30.270751+0000","last_clean_scrub_stamp":"2026-02-18T06:20:30.270751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:33:35.594020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040074499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":425,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.584866+0000","last_change":"2026-02-18T06:20:28.711581+0000","last_active":"2026-02-18T06:44:09.584866+0000","last_peered":"2026-02-18T06:44:09.584866+0000","last_clean":"2026-02-18T06:44:09.584866+0000","last_became_active":"2026-02-18T06:20:28.710726+0000","last_became_peered":"2026-02-18T06:20:28.710726+0000","last_unstale":"2026-02-18T06:44:09.584866+0000","last_undegraded":"2026-02-18T06:44:09.584866+0000","last_fullsized":"2026-02-18T06:44:09.584866+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:27.265730+0000","last_clean_scrub_stamp":"2026-02-18T06:20:27.265730+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T09:48:59.040538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738657,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":262848,"kb_used_data":1392,"kb_used_omap":799,"kb_used_meta":260640,"kb_avail":94108992,"statfs":{"total":96636764160,"available":96367607808,"internally_reserved":0,"allocated":1425408,"data_stored":714295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":818782,"internal_metadata":266895778},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738656,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263432,"kb_used_data":1976,"kb_used_omap":494,"kb_used_meta":260945,"kb_avail":94108408,"statfs":{"total":96636764160,"available":96367009792,"internally_reserved":0,"allocated":2023424,"data_stored":1304682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":505992,"internal_metadata":267208568},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738656,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299528,"kb_used_data":1976,"kb_used_omap":714,"kb_used_meta":296821,"kb_avail":94072312,"statfs":{"total":96636764160,"available":96330047488,"internally_reserved":0,"allocated":2023424,"data_stored":1304682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":732113,"internal_metadata":303944751},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:13.704 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.704 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.705 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.705 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.705 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.705 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.705 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.705 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-18T06:20:30.270751+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=30, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.706 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-18T06:20:27.265730+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=27, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=11, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:13.706 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-18T06:44:33.707 DEBUG:teuthology.orchestra.run.trial130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-18T06:44:33.888 INFO:teuthology.orchestra.run.trial130.stdout: 2026-02-18T06:44:33.889 INFO:teuthology.orchestra.run.trial130.stderr:dumped all 2026-02-18T06:44:33.898 INFO:teuthology.orchestra.run.trial130.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2026-02-18T06:44:33.341963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":825808,"kb_used_data":5344,"kb_used_omap":2008,"kb_used_meta":818407,"kb_avail":282289712,"statfs":{"total":289910292480,"available":289064665088,"internally_reserved":0,"allocated":5472256,"data_stored":3323659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2056887,"internal_metadata":838049097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":342,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:17.373038+0000","last_change":"2026-02-18T06:44:17.372984+0000","last_active":"2026-02-18T06:44:17.373038+0000","last_peered":"2026-02-18T06:44:17.373038+0000","last_clean":"2026-02-18T06:44:17.373038+0000","last_became_active":"2026-02-18T06:20:31.279056+0000","last_became_peered":"2026-02-18T06:20:31.279056+0000","last_unstale":"2026-02-18T06:44:17.373038+0000","last_undegraded":"2026-02-18T06:44:17.373038+0000","last_fullsized":"2026-02-18T06:44:17.373038+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:17.372928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:17.372928+0000","last_clean_scrub_stamp":"2026-02-18T06:44:17.372928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:01:34.063886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053659699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":342,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:18.361320+0000","last_change":"2026-02-18T06:44:18.361300+0000","last_active":"2026-02-18T06:44:18.361320+0000","last_peered":"2026-02-18T06:44:18.361320+0000","last_clean":"2026-02-18T06:44:18.361320+0000","last_became_active":"2026-02-18T06:20:31.279113+0000","last_became_peered":"2026-02-18T06:20:31.279113+0000","last_unstale":"2026-02-18T06:44:18.361320+0000","last_undegraded":"2026-02-18T06:44:18.361320+0000","last_fullsized":"2026-02-18T06:44:18.361320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:18.361264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:18.361264+0000","last_clean_scrub_stamp":"2026-02-18T06:44:18.361264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:27:04.622651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030664000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":342,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:15.347871+0000","last_change":"2026-02-18T06:44:15.347871+0000","last_active":"2026-02-18T06:44:15.347871+0000","last_peered":"2026-02-18T06:44:15.347871+0000","last_clean":"2026-02-18T06:44:15.347871+0000","last_became_active":"2026-02-18T06:20:31.279035+0000","last_became_peered":"2026-02-18T06:20:31.279035+0000","last_unstale":"2026-02-18T06:44:15.347871+0000","last_undegraded":"2026-02-18T06:44:15.347871+0000","last_fullsized":"2026-02-18T06:44:15.347871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:15.347835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:15.347835+0000","last_clean_scrub_stamp":"2026-02-18T06:44:15.347835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T18:09:28.156962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043416700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":342,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:14.366147+0000","last_change":"2026-02-18T06:44:14.366147+0000","last_active":"2026-02-18T06:44:14.366147+0000","last_peered":"2026-02-18T06:44:14.366147+0000","last_clean":"2026-02-18T06:44:14.366147+0000","last_became_active":"2026-02-18T06:20:31.278965+0000","last_became_peered":"2026-02-18T06:20:31.278965+0000","last_unstale":"2026-02-18T06:44:14.366147+0000","last_undegraded":"2026-02-18T06:44:14.366147+0000","last_fullsized":"2026-02-18T06:44:14.366147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:14.366073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:14.366073+0000","last_clean_scrub_stamp":"2026-02-18T06:44:14.366073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:46:30.710785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00027168500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"167'3","reported_seq":346,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:13.153704+0000","last_change":"2026-02-18T06:44:13.153704+0000","last_active":"2026-02-18T06:44:13.153704+0000","last_peered":"2026-02-18T06:44:13.153704+0000","last_clean":"2026-02-18T06:44:13.153704+0000","last_became_active":"2026-02-18T06:20:31.279064+0000","last_became_peered":"2026-02-18T06:20:31.279064+0000","last_unstale":"2026-02-18T06:44:13.153704+0000","last_undegraded":"2026-02-18T06:44:13.153704+0000","last_fullsized":"2026-02-18T06:44:13.153704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'3","last_scrub_stamp":"2026-02-18T06:44:13.153662+0000","last_deep_scrub":"167'3","last_deep_scrub_stamp":"2026-02-18T06:44:13.153662+0000","last_clean_scrub_stamp":"2026-02-18T06:44:13.153662+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:38:22.948044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00020610299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":341,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:13.774945+0000","last_change":"2026-02-18T06:44:13.774945+0000","last_active":"2026-02-18T06:44:13.774945+0000","last_peered":"2026-02-18T06:44:13.774945+0000","last_clean":"2026-02-18T06:44:13.774945+0000","last_became_active":"2026-02-18T06:20:31.279036+0000","last_became_peered":"2026-02-18T06:20:31.279036+0000","last_unstale":"2026-02-18T06:44:13.774945+0000","last_undegraded":"2026-02-18T06:44:13.774945+0000","last_fullsized":"2026-02-18T06:44:13.774945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:13.774894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:13.774894+0000","last_clean_scrub_stamp":"2026-02-18T06:44:13.774894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:16:52.059516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00017900300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":341,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:14.762652+0000","last_change":"2026-02-18T06:44:14.762652+0000","last_active":"2026-02-18T06:44:14.762652+0000","last_peered":"2026-02-18T06:44:14.762652+0000","last_clean":"2026-02-18T06:44:14.762652+0000","last_became_active":"2026-02-18T06:20:31.279067+0000","last_became_peered":"2026-02-18T06:20:31.279067+0000","last_unstale":"2026-02-18T06:44:14.762652+0000","last_undegraded":"2026-02-18T06:44:14.762652+0000","last_fullsized":"2026-02-18T06:44:14.762652+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:14.762603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:14.762603+0000","last_clean_scrub_stamp":"2026-02-18T06:44:14.762603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:21:26.094656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017421299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"167'2","reported_seq":344,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:16.348119+0000","last_change":"2026-02-18T06:44:16.348119+0000","last_active":"2026-02-18T06:44:16.348119+0000","last_peered":"2026-02-18T06:44:16.348119+0000","last_clean":"2026-02-18T06:44:16.348119+0000","last_became_active":"2026-02-18T06:20:31.278647+0000","last_became_peered":"2026-02-18T06:20:31.278647+0000","last_unstale":"2026-02-18T06:44:16.348119+0000","last_undegraded":"2026-02-18T06:44:16.348119+0000","last_fullsized":"2026-02-18T06:44:16.348119+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'2","last_scrub_stamp":"2026-02-18T06:44:16.348091+0000","last_deep_scrub":"167'2","last_deep_scrub_stamp":"2026-02-18T06:44:16.348091+0000","last_clean_scrub_stamp":"2026-02-18T06:44:16.348091+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:49:04.941186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00040074499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"167'74","reported_seq":435,"reported_epoch":167,"state":"active+clean","last_fresh":"2026-02-18T06:44:13.349451+0000","last_change":"2026-02-18T06:44:13.349451+0000","last_active":"2026-02-18T06:44:13.349451+0000","last_peered":"2026-02-18T06:44:13.349451+0000","last_clean":"2026-02-18T06:44:13.349451+0000","last_became_active":"2026-02-18T06:20:28.710726+0000","last_became_peered":"2026-02-18T06:20:28.710726+0000","last_unstale":"2026-02-18T06:44:13.349451+0000","last_undegraded":"2026-02-18T06:44:13.349451+0000","last_fullsized":"2026-02-18T06:44:13.349451+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'74","last_scrub_stamp":"2026-02-18T06:44:13.349400+0000","last_deep_scrub":"167'74","last_deep_scrub_stamp":"2026-02-18T06:44:13.349400+0000","last_clean_scrub_stamp":"2026-02-18T06:44:13.349400+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:02:01.181838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":262848,"kb_used_data":1392,"kb_used_omap":799,"kb_used_meta":260640,"kb_avail":94108992,"statfs":{"total":96636764160,"available":96367607808,"internally_reserved":0,"allocated":1425408,"data_stored":714295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":818782,"internal_metadata":266895778},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738660,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263432,"kb_used_data":1976,"kb_used_omap":494,"kb_used_meta":260945,"kb_avail":94108408,"statfs":{"total":96636764160,"available":96367009792,"internally_reserved":0,"allocated":2023424,"data_stored":1304682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":505992,"internal_metadata":267208568},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299528,"kb_used_data":1976,"kb_used_omap":714,"kb_used_meta":296821,"kb_avail":94072312,"statfs":{"total":96636764160,"available":96330047488,"internally_reserved":0,"allocated":2023424,"data_stored":1304682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":732113,"internal_metadata":303944751},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:33.900 DEBUG:teuthology.orchestra.run.trial130:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2026-02-18T06:44:34.099 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-18T06:44:34.099 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-18T06:44:34.100 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2026-02-18T06:44:34.100 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:44:34.147 INFO:tasks.ceph.cluster2.osd.0:Stopped 2026-02-18T06:44:34.147 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2026-02-18T06:44:34.147 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:44:34.194 INFO:tasks.ceph.cluster2.osd.1:Stopped 2026-02-18T06:44:34.194 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2026-02-18T06:44:34.194 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:44:34.240 INFO:tasks.ceph.cluster2.osd.2:Stopped 2026-02-18T06:44:34.241 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-18T06:44:34.241 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2026-02-18T06:44:34.241 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:44:35.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.mgr.x is failed for ~0s 2026-02-18T06:44:40.343 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2026-02-18T06:44:40.343 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-18T06:44:40.343 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2026-02-18T06:44:40.343 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:44:40.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.mon.a is failed for ~0s 2026-02-18T06:44:40.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.mgr.x has been restored 2026-02-18T06:44:40.357 INFO:tasks.ceph.cluster2.mon.a:Stopped 2026-02-18T06:44:40.357 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-18T06:44:40.357 DEBUG:teuthology.orchestra.run.trial130:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-18T06:44:40.422 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:44:40.422 DEBUG:teuthology.orchestra.run.trial130:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2026-02-18T06:44:40.502 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:44:40.502 DEBUG:teuthology.orchestra.run.trial130:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2026-02-18T06:44:40.587 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:44:40.587 DEBUG:teuthology.orchestra.run.trial130:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2026-02-18T06:44:40.632 INFO:tasks.ceph:Archiving mon data... 2026-02-18T06:44:40.634 DEBUG:teuthology.misc:Transferring archived files from trial130:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/data/cluster2.mon.a.tgz 2026-02-18T06:44:40.634 DEBUG:teuthology.orchestra.run.trial130:> mktemp 2026-02-18T06:44:40.650 INFO:teuthology.orchestra.run.trial130.stdout:/tmp/tmp.c7vciQUjNo 2026-02-18T06:44:40.650 DEBUG:teuthology.orchestra.run.trial130:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.c7vciQUjNo 2026-02-18T06:44:40.766 DEBUG:teuthology.orchestra.run.trial130:> sudo chmod 0666 /tmp/tmp.c7vciQUjNo 2026-02-18T06:44:40.845 DEBUG:teuthology.orchestra.remote:trial130:/tmp/tmp.c7vciQUjNo is 423KB 2026-02-18T06:44:40.935 DEBUG:teuthology.orchestra.run.trial130:> rm -fr /tmp/tmp.c7vciQUjNo 2026-02-18T06:44:40.951 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-18T06:44:40.952 DEBUG:teuthology.orchestra.run.trial031:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-18T06:44:40.954 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-18T06:44:41.015 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-18T06:44:41.028 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-18T06:44:41.029 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-18T06:44:41.209 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:44:41.209 INFO:teuthology.orchestra.run.trial031.stderr:dumped all 2026-02-18T06:44:41.218 INFO:teuthology.orchestra.run.trial031.stdout:{"pg_ready":true,"pg_map":{"version":918,"stamp":"2026-02-18T06:44:39.732630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":411292,"kb_used_data":5740,"kb_used_omap":1914,"kb_used_meta":403589,"kb_avail":282704228,"statfs":{"total":289910292480,"available":289489129472,"internally_reserved":0,"allocated":5877760,"data_stored":3537691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1960828,"internal_metadata":413275268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000944"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240773+0000","last_change":"2026-02-18T06:20:10.545651+0000","last_active":"2026-02-18T06:44:09.240773+0000","last_peered":"2026-02-18T06:44:09.240773+0000","last_clean":"2026-02-18T06:44:09.240773+0000","last_became_active":"2026-02-18T06:20:08.542218+0000","last_became_peered":"2026-02-18T06:20:08.542218+0000","last_unstale":"2026-02-18T06:44:09.240773+0000","last_undegraded":"2026-02-18T06:44:09.240773+0000","last_fullsized":"2026-02-18T06:44:09.240773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:04:54.990306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095951000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.241404+0000","last_change":"2026-02-18T06:20:10.545246+0000","last_active":"2026-02-18T06:44:09.241404+0000","last_peered":"2026-02-18T06:44:09.241404+0000","last_clean":"2026-02-18T06:44:09.241404+0000","last_became_active":"2026-02-18T06:20:08.541842+0000","last_became_peered":"2026-02-18T06:20:08.541842+0000","last_unstale":"2026-02-18T06:44:09.241404+0000","last_undegraded":"2026-02-18T06:44:09.241404+0000","last_fullsized":"2026-02-18T06:44:09.241404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:21:47.699026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059815900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.242141+0000","last_change":"2026-02-18T06:20:10.545338+0000","last_active":"2026-02-18T06:44:09.242141+0000","last_peered":"2026-02-18T06:44:09.242141+0000","last_clean":"2026-02-18T06:44:09.242141+0000","last_became_active":"2026-02-18T06:20:08.541791+0000","last_became_peered":"2026-02-18T06:20:08.541791+0000","last_unstale":"2026-02-18T06:44:09.242141+0000","last_undegraded":"2026-02-18T06:44:09.242141+0000","last_fullsized":"2026-02-18T06:44:09.242141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:57:01.887084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068669399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240212+0000","last_change":"2026-02-18T06:20:10.545063+0000","last_active":"2026-02-18T06:44:09.240212+0000","last_peered":"2026-02-18T06:44:09.240212+0000","last_clean":"2026-02-18T06:44:09.240212+0000","last_became_active":"2026-02-18T06:20:08.541914+0000","last_became_peered":"2026-02-18T06:20:08.541914+0000","last_unstale":"2026-02-18T06:44:09.240212+0000","last_undegraded":"2026-02-18T06:44:09.240212+0000","last_fullsized":"2026-02-18T06:44:09.240212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:35:47.720493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408427,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087416+0000","last_change":"2026-02-18T06:20:10.543913+0000","last_active":"2026-02-18T06:44:10.087416+0000","last_peered":"2026-02-18T06:44:10.087416+0000","last_clean":"2026-02-18T06:44:10.087416+0000","last_became_active":"2026-02-18T06:20:08.541913+0000","last_became_peered":"2026-02-18T06:20:08.541913+0000","last_unstale":"2026-02-18T06:44:10.087416+0000","last_undegraded":"2026-02-18T06:44:10.087416+0000","last_fullsized":"2026-02-18T06:44:10.087416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:01:29.992866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077138100000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.088825+0000","last_change":"2026-02-18T06:20:10.739368+0000","last_active":"2026-02-18T06:44:10.088825+0000","last_peered":"2026-02-18T06:44:10.088825+0000","last_clean":"2026-02-18T06:44:10.088825+0000","last_became_active":"2026-02-18T06:20:08.541846+0000","last_became_peered":"2026-02-18T06:20:08.541846+0000","last_unstale":"2026-02-18T06:44:10.088825+0000","last_undegraded":"2026-02-18T06:44:10.088825+0000","last_fullsized":"2026-02-18T06:44:10.088825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:19:18.113602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000219657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087287+0000","last_change":"2026-02-18T06:20:10.739337+0000","last_active":"2026-02-18T06:44:10.087287+0000","last_peered":"2026-02-18T06:44:10.087287+0000","last_clean":"2026-02-18T06:44:10.087287+0000","last_became_active":"2026-02-18T06:20:08.541956+0000","last_became_peered":"2026-02-18T06:20:08.541956+0000","last_unstale":"2026-02-18T06:44:10.087287+0000","last_undegraded":"2026-02-18T06:44:10.087287+0000","last_fullsized":"2026-02-18T06:44:10.087287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:06:02.012849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023866299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240905+0000","last_change":"2026-02-18T06:20:10.544993+0000","last_active":"2026-02-18T06:44:09.240905+0000","last_peered":"2026-02-18T06:44:09.240905+0000","last_clean":"2026-02-18T06:44:09.240905+0000","last_became_active":"2026-02-18T06:20:08.541386+0000","last_became_peered":"2026-02-18T06:20:08.541386+0000","last_unstale":"2026-02-18T06:44:09.240905+0000","last_undegraded":"2026-02-18T06:44:09.240905+0000","last_fullsized":"2026-02-18T06:44:09.240905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:11:31.968149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034991900000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":457,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.238991+0000","last_change":"2026-02-18T06:20:05.694236+0000","last_active":"2026-02-18T06:44:09.238991+0000","last_peered":"2026-02-18T06:44:09.238991+0000","last_clean":"2026-02-18T06:44:09.238991+0000","last_became_active":"2026-02-18T06:20:05.693589+0000","last_became_peered":"2026-02-18T06:20:05.693589+0000","last_unstale":"2026-02-18T06:44:09.238991+0000","last_undegraded":"2026-02-18T06:44:09.238991+0000","last_fullsized":"2026-02-18T06:44:09.238991+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_clean_scrub_stamp":"2026-02-18T06:20:04.531394+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:48:22.870932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738665,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136708,"kb_used_data":1524,"kb_used_omap":701,"kb_used_meta":134466,"kb_avail":94235132,"statfs":{"total":96636764160,"available":96496775168,"internally_reserved":0,"allocated":1560576,"data_stored":785639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718806,"internal_metadata":137693226},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":464,"kb_used_meta":134703,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475729,"internal_metadata":137936303},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":748,"kb_used_meta":134419,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":766293,"internal_metadata":137645739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:41.220 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-18T06:44:41.400 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:44:41.401 INFO:teuthology.orchestra.run.trial031.stderr:dumped all 2026-02-18T06:44:41.410 INFO:teuthology.orchestra.run.trial031.stdout:{"pg_ready":true,"pg_map":{"version":918,"stamp":"2026-02-18T06:44:39.732630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":411292,"kb_used_data":5740,"kb_used_omap":1914,"kb_used_meta":403589,"kb_avail":282704228,"statfs":{"total":289910292480,"available":289489129472,"internally_reserved":0,"allocated":5877760,"data_stored":3537691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1960828,"internal_metadata":413275268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000944"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240773+0000","last_change":"2026-02-18T06:20:10.545651+0000","last_active":"2026-02-18T06:44:09.240773+0000","last_peered":"2026-02-18T06:44:09.240773+0000","last_clean":"2026-02-18T06:44:09.240773+0000","last_became_active":"2026-02-18T06:20:08.542218+0000","last_became_peered":"2026-02-18T06:20:08.542218+0000","last_unstale":"2026-02-18T06:44:09.240773+0000","last_undegraded":"2026-02-18T06:44:09.240773+0000","last_fullsized":"2026-02-18T06:44:09.240773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:04:54.990306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095951000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.241404+0000","last_change":"2026-02-18T06:20:10.545246+0000","last_active":"2026-02-18T06:44:09.241404+0000","last_peered":"2026-02-18T06:44:09.241404+0000","last_clean":"2026-02-18T06:44:09.241404+0000","last_became_active":"2026-02-18T06:20:08.541842+0000","last_became_peered":"2026-02-18T06:20:08.541842+0000","last_unstale":"2026-02-18T06:44:09.241404+0000","last_undegraded":"2026-02-18T06:44:09.241404+0000","last_fullsized":"2026-02-18T06:44:09.241404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:21:47.699026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059815900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.242141+0000","last_change":"2026-02-18T06:20:10.545338+0000","last_active":"2026-02-18T06:44:09.242141+0000","last_peered":"2026-02-18T06:44:09.242141+0000","last_clean":"2026-02-18T06:44:09.242141+0000","last_became_active":"2026-02-18T06:20:08.541791+0000","last_became_peered":"2026-02-18T06:20:08.541791+0000","last_unstale":"2026-02-18T06:44:09.242141+0000","last_undegraded":"2026-02-18T06:44:09.242141+0000","last_fullsized":"2026-02-18T06:44:09.242141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:57:01.887084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068669399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240212+0000","last_change":"2026-02-18T06:20:10.545063+0000","last_active":"2026-02-18T06:44:09.240212+0000","last_peered":"2026-02-18T06:44:09.240212+0000","last_clean":"2026-02-18T06:44:09.240212+0000","last_became_active":"2026-02-18T06:20:08.541914+0000","last_became_peered":"2026-02-18T06:20:08.541914+0000","last_unstale":"2026-02-18T06:44:09.240212+0000","last_undegraded":"2026-02-18T06:44:09.240212+0000","last_fullsized":"2026-02-18T06:44:09.240212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:35:47.720493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408427,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087416+0000","last_change":"2026-02-18T06:20:10.543913+0000","last_active":"2026-02-18T06:44:10.087416+0000","last_peered":"2026-02-18T06:44:10.087416+0000","last_clean":"2026-02-18T06:44:10.087416+0000","last_became_active":"2026-02-18T06:20:08.541913+0000","last_became_peered":"2026-02-18T06:20:08.541913+0000","last_unstale":"2026-02-18T06:44:10.087416+0000","last_undegraded":"2026-02-18T06:44:10.087416+0000","last_fullsized":"2026-02-18T06:44:10.087416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:01:29.992866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077138100000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.088825+0000","last_change":"2026-02-18T06:20:10.739368+0000","last_active":"2026-02-18T06:44:10.088825+0000","last_peered":"2026-02-18T06:44:10.088825+0000","last_clean":"2026-02-18T06:44:10.088825+0000","last_became_active":"2026-02-18T06:20:08.541846+0000","last_became_peered":"2026-02-18T06:20:08.541846+0000","last_unstale":"2026-02-18T06:44:10.088825+0000","last_undegraded":"2026-02-18T06:44:10.088825+0000","last_fullsized":"2026-02-18T06:44:10.088825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:19:18.113602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000219657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087287+0000","last_change":"2026-02-18T06:20:10.739337+0000","last_active":"2026-02-18T06:44:10.087287+0000","last_peered":"2026-02-18T06:44:10.087287+0000","last_clean":"2026-02-18T06:44:10.087287+0000","last_became_active":"2026-02-18T06:20:08.541956+0000","last_became_peered":"2026-02-18T06:20:08.541956+0000","last_unstale":"2026-02-18T06:44:10.087287+0000","last_undegraded":"2026-02-18T06:44:10.087287+0000","last_fullsized":"2026-02-18T06:44:10.087287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:06:02.012849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023866299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240905+0000","last_change":"2026-02-18T06:20:10.544993+0000","last_active":"2026-02-18T06:44:09.240905+0000","last_peered":"2026-02-18T06:44:09.240905+0000","last_clean":"2026-02-18T06:44:09.240905+0000","last_became_active":"2026-02-18T06:20:08.541386+0000","last_became_peered":"2026-02-18T06:20:08.541386+0000","last_unstale":"2026-02-18T06:44:09.240905+0000","last_undegraded":"2026-02-18T06:44:09.240905+0000","last_fullsized":"2026-02-18T06:44:09.240905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:11:31.968149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034991900000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":457,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.238991+0000","last_change":"2026-02-18T06:20:05.694236+0000","last_active":"2026-02-18T06:44:09.238991+0000","last_peered":"2026-02-18T06:44:09.238991+0000","last_clean":"2026-02-18T06:44:09.238991+0000","last_became_active":"2026-02-18T06:20:05.693589+0000","last_became_peered":"2026-02-18T06:20:05.693589+0000","last_unstale":"2026-02-18T06:44:09.238991+0000","last_undegraded":"2026-02-18T06:44:09.238991+0000","last_fullsized":"2026-02-18T06:44:09.238991+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_clean_scrub_stamp":"2026-02-18T06:20:04.531394+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:48:22.870932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738665,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136708,"kb_used_data":1524,"kb_used_omap":701,"kb_used_meta":134466,"kb_avail":94235132,"statfs":{"total":96636764160,"available":96496775168,"internally_reserved":0,"allocated":1560576,"data_stored":785639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718806,"internal_metadata":137693226},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":464,"kb_used_meta":134703,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475729,"internal_metadata":137936303},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":748,"kb_used_meta":134419,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":766293,"internal_metadata":137645739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:41.412 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-18T06:44:41.412 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-18T06:44:41.591 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:44:41.591 INFO:teuthology.orchestra.run.trial031.stderr:dumped all 2026-02-18T06:44:41.599 INFO:teuthology.orchestra.run.trial031.stdout:{"pg_ready":true,"pg_map":{"version":918,"stamp":"2026-02-18T06:44:39.732630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":411292,"kb_used_data":5740,"kb_used_omap":1914,"kb_used_meta":403589,"kb_avail":282704228,"statfs":{"total":289910292480,"available":289489129472,"internally_reserved":0,"allocated":5877760,"data_stored":3537691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1960828,"internal_metadata":413275268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000944"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240773+0000","last_change":"2026-02-18T06:20:10.545651+0000","last_active":"2026-02-18T06:44:09.240773+0000","last_peered":"2026-02-18T06:44:09.240773+0000","last_clean":"2026-02-18T06:44:09.240773+0000","last_became_active":"2026-02-18T06:20:08.542218+0000","last_became_peered":"2026-02-18T06:20:08.542218+0000","last_unstale":"2026-02-18T06:44:09.240773+0000","last_undegraded":"2026-02-18T06:44:09.240773+0000","last_fullsized":"2026-02-18T06:44:09.240773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:04:54.990306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095951000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.241404+0000","last_change":"2026-02-18T06:20:10.545246+0000","last_active":"2026-02-18T06:44:09.241404+0000","last_peered":"2026-02-18T06:44:09.241404+0000","last_clean":"2026-02-18T06:44:09.241404+0000","last_became_active":"2026-02-18T06:20:08.541842+0000","last_became_peered":"2026-02-18T06:20:08.541842+0000","last_unstale":"2026-02-18T06:44:09.241404+0000","last_undegraded":"2026-02-18T06:44:09.241404+0000","last_fullsized":"2026-02-18T06:44:09.241404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:21:47.699026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059815900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.242141+0000","last_change":"2026-02-18T06:20:10.545338+0000","last_active":"2026-02-18T06:44:09.242141+0000","last_peered":"2026-02-18T06:44:09.242141+0000","last_clean":"2026-02-18T06:44:09.242141+0000","last_became_active":"2026-02-18T06:20:08.541791+0000","last_became_peered":"2026-02-18T06:20:08.541791+0000","last_unstale":"2026-02-18T06:44:09.242141+0000","last_undegraded":"2026-02-18T06:44:09.242141+0000","last_fullsized":"2026-02-18T06:44:09.242141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:57:01.887084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068669399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240212+0000","last_change":"2026-02-18T06:20:10.545063+0000","last_active":"2026-02-18T06:44:09.240212+0000","last_peered":"2026-02-18T06:44:09.240212+0000","last_clean":"2026-02-18T06:44:09.240212+0000","last_became_active":"2026-02-18T06:20:08.541914+0000","last_became_peered":"2026-02-18T06:20:08.541914+0000","last_unstale":"2026-02-18T06:44:09.240212+0000","last_undegraded":"2026-02-18T06:44:09.240212+0000","last_fullsized":"2026-02-18T06:44:09.240212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:35:47.720493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408427,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087416+0000","last_change":"2026-02-18T06:20:10.543913+0000","last_active":"2026-02-18T06:44:10.087416+0000","last_peered":"2026-02-18T06:44:10.087416+0000","last_clean":"2026-02-18T06:44:10.087416+0000","last_became_active":"2026-02-18T06:20:08.541913+0000","last_became_peered":"2026-02-18T06:20:08.541913+0000","last_unstale":"2026-02-18T06:44:10.087416+0000","last_undegraded":"2026-02-18T06:44:10.087416+0000","last_fullsized":"2026-02-18T06:44:10.087416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:01:29.992866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077138100000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.088825+0000","last_change":"2026-02-18T06:20:10.739368+0000","last_active":"2026-02-18T06:44:10.088825+0000","last_peered":"2026-02-18T06:44:10.088825+0000","last_clean":"2026-02-18T06:44:10.088825+0000","last_became_active":"2026-02-18T06:20:08.541846+0000","last_became_peered":"2026-02-18T06:20:08.541846+0000","last_unstale":"2026-02-18T06:44:10.088825+0000","last_undegraded":"2026-02-18T06:44:10.088825+0000","last_fullsized":"2026-02-18T06:44:10.088825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:19:18.113602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000219657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087287+0000","last_change":"2026-02-18T06:20:10.739337+0000","last_active":"2026-02-18T06:44:10.087287+0000","last_peered":"2026-02-18T06:44:10.087287+0000","last_clean":"2026-02-18T06:44:10.087287+0000","last_became_active":"2026-02-18T06:20:08.541956+0000","last_became_peered":"2026-02-18T06:20:08.541956+0000","last_unstale":"2026-02-18T06:44:10.087287+0000","last_undegraded":"2026-02-18T06:44:10.087287+0000","last_fullsized":"2026-02-18T06:44:10.087287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:06:02.012849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023866299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240905+0000","last_change":"2026-02-18T06:20:10.544993+0000","last_active":"2026-02-18T06:44:09.240905+0000","last_peered":"2026-02-18T06:44:09.240905+0000","last_clean":"2026-02-18T06:44:09.240905+0000","last_became_active":"2026-02-18T06:20:08.541386+0000","last_became_peered":"2026-02-18T06:20:08.541386+0000","last_unstale":"2026-02-18T06:44:09.240905+0000","last_undegraded":"2026-02-18T06:44:09.240905+0000","last_fullsized":"2026-02-18T06:44:09.240905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:11:31.968149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034991900000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":457,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.238991+0000","last_change":"2026-02-18T06:20:05.694236+0000","last_active":"2026-02-18T06:44:09.238991+0000","last_peered":"2026-02-18T06:44:09.238991+0000","last_clean":"2026-02-18T06:44:09.238991+0000","last_became_active":"2026-02-18T06:20:05.693589+0000","last_became_peered":"2026-02-18T06:20:05.693589+0000","last_unstale":"2026-02-18T06:44:09.238991+0000","last_undegraded":"2026-02-18T06:44:09.238991+0000","last_fullsized":"2026-02-18T06:44:09.238991+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_clean_scrub_stamp":"2026-02-18T06:20:04.531394+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:48:22.870932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738665,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136708,"kb_used_data":1524,"kb_used_omap":701,"kb_used_meta":134466,"kb_avail":94235132,"statfs":{"total":96636764160,"available":96496775168,"internally_reserved":0,"allocated":1560576,"data_stored":785639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718806,"internal_metadata":137693226},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":464,"kb_used_meta":134703,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475729,"internal_metadata":137936303},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":748,"kb_used_meta":134419,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":766293,"internal_metadata":137645739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:41.601 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-18T06:44:41.783 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:44:41.783 INFO:teuthology.orchestra.run.trial031.stdout:{"epoch":183,"fsid":"c34c7f09-e418-4318-892f-b221bab3e02e","created":"2026-02-18T06:19:59.102474+0000","modified":"2026-02-18T06:44:09.235005+0000","last_up_change":"2026-02-18T06:20:03.109548+0000","last_in_change":"2026-02-18T06:19:59.789405+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-18T06:20:03.538981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-18T06:20:07.190513+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"269a8c5d-3e3d-44c0-969c-686555b67a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6808","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6809","nonce":2229877145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6810","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6811","nonce":2229877145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6814","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6815","nonce":2229877145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6812","nonce":2229877145},{"type":"v1","addr":"10.20.193.31:6813","nonce":2229877145}]},"public_addr":"10.20.193.31:6809/2229877145","cluster_addr":"10.20.193.31:6811/2229877145","heartbeat_back_addr":"10.20.193.31:6815/2229877145","heartbeat_front_addr":"10.20.193.31:6813/2229877145","state":["exists","up"]},{"osd":1,"uuid":"91fe9ddc-6f8a-4a7f-ae09-18bea6a1597b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6800","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6801","nonce":1524508517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6802","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6803","nonce":1524508517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6806","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6807","nonce":1524508517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6804","nonce":1524508517},{"type":"v1","addr":"10.20.193.31:6805","nonce":1524508517}]},"public_addr":"10.20.193.31:6801/1524508517","cluster_addr":"10.20.193.31:6803/1524508517","heartbeat_back_addr":"10.20.193.31:6807/1524508517","heartbeat_front_addr":"10.20.193.31:6805/1524508517","state":["exists","up"]},{"osd":2,"uuid":"d87b9373-e7c3-4a6e-b065-c5ee208d646f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6816","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6817","nonce":1969377339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6818","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6819","nonce":1969377339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6822","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6823","nonce":1969377339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.31:6820","nonce":1969377339},{"type":"v1","addr":"10.20.193.31:6821","nonce":1969377339}]},"public_addr":"10.20.193.31:6817/1969377339","cluster_addr":"10.20.193.31:6819/1969377339","heartbeat_back_addr":"10.20.193.31:6823/1969377339","heartbeat_front_addr":"10.20.193.31:6821/1969377339","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-18T06:20:01.800668+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-18T06:20:01.671424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-18T06:20:01.767795+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.130:0/933722940":"2026-02-18T07:43:54.670982+0000","10.20.193.130:0/3874574642":"2026-02-18T07:43:54.425083+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":49,"length":3}]},{"pool":5,"snaps":[{"begin":3,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-18T06:44:42.794 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2026-02-18T06:44:42.794 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-18T06:44:42.877 INFO:teuthology.orchestra.run.trial031.stdout:{ 2026-02-18T06:44:42.877 INFO:teuthology.orchestra.run.trial031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-18T06:44:42.877 INFO:teuthology.orchestra.run.trial031.stdout:} 2026-02-18T06:44:42.885 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2026-02-18T06:44:43.067 INFO:teuthology.orchestra.run.trial031.stderr:instructed osd(s) 0 to deep-scrub 2026-02-18T06:44:43.078 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2026-02-18T06:44:43.078 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-18T06:44:43.158 INFO:teuthology.orchestra.run.trial031.stdout:{ 2026-02-18T06:44:43.158 INFO:teuthology.orchestra.run.trial031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-18T06:44:43.159 INFO:teuthology.orchestra.run.trial031.stdout:} 2026-02-18T06:44:43.164 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2026-02-18T06:44:43.343 INFO:teuthology.orchestra.run.trial031.stderr:instructed osd(s) 1 to deep-scrub 2026-02-18T06:44:43.353 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2026-02-18T06:44:43.353 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-18T06:44:43.435 INFO:teuthology.orchestra.run.trial031.stdout:{ 2026-02-18T06:44:43.435 INFO:teuthology.orchestra.run.trial031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-18T06:44:43.435 INFO:teuthology.orchestra.run.trial031.stdout:} 2026-02-18T06:44:43.442 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2026-02-18T06:44:43.623 INFO:teuthology.orchestra.run.trial031.stderr:instructed osd(s) 2 to deep-scrub 2026-02-18T06:44:43.632 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-18T06:44:43.817 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:44:43.817 INFO:teuthology.orchestra.run.trial031.stderr:dumped all 2026-02-18T06:44:43.826 INFO:teuthology.orchestra.run.trial031.stdout:{"pg_ready":true,"pg_map":{"version":920,"stamp":"2026-02-18T06:44:43.733054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":411292,"kb_used_data":5740,"kb_used_omap":1914,"kb_used_meta":403589,"kb_avail":282704228,"statfs":{"total":289910292480,"available":289489129472,"internally_reserved":0,"allocated":5877760,"data_stored":3537691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1960828,"internal_metadata":413275268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001102"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240773+0000","last_change":"2026-02-18T06:20:10.545651+0000","last_active":"2026-02-18T06:44:09.240773+0000","last_peered":"2026-02-18T06:44:09.240773+0000","last_clean":"2026-02-18T06:44:09.240773+0000","last_became_active":"2026-02-18T06:20:08.542218+0000","last_became_peered":"2026-02-18T06:20:08.542218+0000","last_unstale":"2026-02-18T06:44:09.240773+0000","last_undegraded":"2026-02-18T06:44:09.240773+0000","last_fullsized":"2026-02-18T06:44:09.240773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:04:54.990306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095951000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.241404+0000","last_change":"2026-02-18T06:20:10.545246+0000","last_active":"2026-02-18T06:44:09.241404+0000","last_peered":"2026-02-18T06:44:09.241404+0000","last_clean":"2026-02-18T06:44:09.241404+0000","last_became_active":"2026-02-18T06:20:08.541842+0000","last_became_peered":"2026-02-18T06:20:08.541842+0000","last_unstale":"2026-02-18T06:44:09.241404+0000","last_undegraded":"2026-02-18T06:44:09.241404+0000","last_fullsized":"2026-02-18T06:44:09.241404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:21:47.699026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059815900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.242141+0000","last_change":"2026-02-18T06:20:10.545338+0000","last_active":"2026-02-18T06:44:09.242141+0000","last_peered":"2026-02-18T06:44:09.242141+0000","last_clean":"2026-02-18T06:44:09.242141+0000","last_became_active":"2026-02-18T06:20:08.541791+0000","last_became_peered":"2026-02-18T06:20:08.541791+0000","last_unstale":"2026-02-18T06:44:09.242141+0000","last_undegraded":"2026-02-18T06:44:09.242141+0000","last_fullsized":"2026-02-18T06:44:09.242141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:57:01.887084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068669399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":366,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240212+0000","last_change":"2026-02-18T06:20:10.545063+0000","last_active":"2026-02-18T06:44:09.240212+0000","last_peered":"2026-02-18T06:44:09.240212+0000","last_clean":"2026-02-18T06:44:09.240212+0000","last_became_active":"2026-02-18T06:20:08.541914+0000","last_became_peered":"2026-02-18T06:20:08.541914+0000","last_unstale":"2026-02-18T06:44:09.240212+0000","last_undegraded":"2026-02-18T06:44:09.240212+0000","last_fullsized":"2026-02-18T06:44:09.240212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:35:47.720493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408427,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087416+0000","last_change":"2026-02-18T06:20:10.543913+0000","last_active":"2026-02-18T06:44:10.087416+0000","last_peered":"2026-02-18T06:44:10.087416+0000","last_clean":"2026-02-18T06:44:10.087416+0000","last_became_active":"2026-02-18T06:20:08.541913+0000","last_became_peered":"2026-02-18T06:20:08.541913+0000","last_unstale":"2026-02-18T06:44:10.087416+0000","last_undegraded":"2026-02-18T06:44:10.087416+0000","last_fullsized":"2026-02-18T06:44:10.087416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:01:29.992866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077138100000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.088825+0000","last_change":"2026-02-18T06:20:10.739368+0000","last_active":"2026-02-18T06:44:10.088825+0000","last_peered":"2026-02-18T06:44:10.088825+0000","last_clean":"2026-02-18T06:44:10.088825+0000","last_became_active":"2026-02-18T06:20:08.541846+0000","last_became_peered":"2026-02-18T06:20:08.541846+0000","last_unstale":"2026-02-18T06:44:10.088825+0000","last_undegraded":"2026-02-18T06:44:10.088825+0000","last_fullsized":"2026-02-18T06:44:10.088825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:19:18.113602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000219657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":368,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:10.087287+0000","last_change":"2026-02-18T06:20:10.739337+0000","last_active":"2026-02-18T06:44:10.087287+0000","last_peered":"2026-02-18T06:44:10.087287+0000","last_clean":"2026-02-18T06:44:10.087287+0000","last_became_active":"2026-02-18T06:20:08.541956+0000","last_became_peered":"2026-02-18T06:20:08.541956+0000","last_unstale":"2026-02-18T06:44:10.087287+0000","last_undegraded":"2026-02-18T06:44:10.087287+0000","last_fullsized":"2026-02-18T06:44:10.087287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:06:02.012849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023866299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":367,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.240905+0000","last_change":"2026-02-18T06:20:10.544993+0000","last_active":"2026-02-18T06:44:09.240905+0000","last_peered":"2026-02-18T06:44:09.240905+0000","last_clean":"2026-02-18T06:44:09.240905+0000","last_became_active":"2026-02-18T06:20:08.541386+0000","last_became_peered":"2026-02-18T06:20:08.541386+0000","last_unstale":"2026-02-18T06:44:09.240905+0000","last_undegraded":"2026-02-18T06:44:09.240905+0000","last_fullsized":"2026-02-18T06:44:09.240905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:07.534525+0000","last_clean_scrub_stamp":"2026-02-18T06:20:07.534525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:11:31.968149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034991900000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":457,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:09.238991+0000","last_change":"2026-02-18T06:20:05.694236+0000","last_active":"2026-02-18T06:44:09.238991+0000","last_peered":"2026-02-18T06:44:09.238991+0000","last_clean":"2026-02-18T06:44:09.238991+0000","last_became_active":"2026-02-18T06:20:05.693589+0000","last_became_peered":"2026-02-18T06:20:05.693589+0000","last_unstale":"2026-02-18T06:44:09.238991+0000","last_undegraded":"2026-02-18T06:44:09.238991+0000","last_fullsized":"2026-02-18T06:44:09.238991+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:20:04.531394+0000","last_clean_scrub_stamp":"2026-02-18T06:20:04.531394+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:48:22.870932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136708,"kb_used_data":1524,"kb_used_omap":701,"kb_used_meta":134466,"kb_avail":94235132,"statfs":{"total":96636764160,"available":96496775168,"internally_reserved":0,"allocated":1560576,"data_stored":785639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718806,"internal_metadata":137693226},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":464,"kb_used_meta":134703,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475729,"internal_metadata":137936303},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":748,"kb_used_meta":134419,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":766293,"internal_metadata":137645739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:44:43.828 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.828 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.828 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.829 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.829 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.829 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.829 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.829 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-18T06:20:07.534525+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=7, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.829 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-18T06:20:04.531394+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=20, tm_sec=4, tm_wday=2, tm_yday=49, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=18, tm_hour=6, tm_min=44, tm_sec=41, tm_wday=2, tm_yday=49, tm_isdst=0) 2026-02-18T06:44:43.830 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-18T06:44:45.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.mon.a has been restored 2026-02-18T06:45:03.831 DEBUG:teuthology.orchestra.run.trial031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-18T06:45:04.010 INFO:teuthology.orchestra.run.trial031.stdout: 2026-02-18T06:45:04.010 INFO:teuthology.orchestra.run.trial031.stderr:dumped all 2026-02-18T06:45:04.019 INFO:teuthology.orchestra.run.trial031.stdout:{"pg_ready":true,"pg_map":{"version":930,"stamp":"2026-02-18T06:45:03.734662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":411292,"kb_used_data":5740,"kb_used_omap":1914,"kb_used_meta":403589,"kb_avail":282704228,"statfs":{"total":289910292480,"available":289489129472,"internally_reserved":0,"allocated":5877760,"data_stored":3537691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1960828,"internal_metadata":413275268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000917"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":374,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:48.730550+0000","last_change":"2026-02-18T06:44:48.730550+0000","last_active":"2026-02-18T06:44:48.730550+0000","last_peered":"2026-02-18T06:44:48.730550+0000","last_clean":"2026-02-18T06:44:48.730550+0000","last_became_active":"2026-02-18T06:20:08.542218+0000","last_became_peered":"2026-02-18T06:20:08.542218+0000","last_unstale":"2026-02-18T06:44:48.730550+0000","last_undegraded":"2026-02-18T06:44:48.730550+0000","last_fullsized":"2026-02-18T06:44:48.730550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:48.730502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:48.730502+0000","last_clean_scrub_stamp":"2026-02-18T06:44:48.730502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T12:36:14.883510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00095951000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":374,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:46.712832+0000","last_change":"2026-02-18T06:44:46.712832+0000","last_active":"2026-02-18T06:44:46.712832+0000","last_peered":"2026-02-18T06:44:46.712832+0000","last_clean":"2026-02-18T06:44:46.712832+0000","last_became_active":"2026-02-18T06:20:08.541842+0000","last_became_peered":"2026-02-18T06:20:08.541842+0000","last_unstale":"2026-02-18T06:44:46.712832+0000","last_undegraded":"2026-02-18T06:44:46.712832+0000","last_fullsized":"2026-02-18T06:44:46.712832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:46.712780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:46.712780+0000","last_clean_scrub_stamp":"2026-02-18T06:44:46.712780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T14:54:54.617838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059815900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":374,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:47.742858+0000","last_change":"2026-02-18T06:44:47.742858+0000","last_active":"2026-02-18T06:44:47.742858+0000","last_peered":"2026-02-18T06:44:47.742858+0000","last_clean":"2026-02-18T06:44:47.742858+0000","last_became_active":"2026-02-18T06:20:08.541791+0000","last_became_peered":"2026-02-18T06:20:08.541791+0000","last_unstale":"2026-02-18T06:44:47.742858+0000","last_undegraded":"2026-02-18T06:44:47.742858+0000","last_fullsized":"2026-02-18T06:44:47.742858+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:47.742818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:47.742818+0000","last_clean_scrub_stamp":"2026-02-18T06:44:47.742818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T16:32:49.331329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068669399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":374,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:45.723516+0000","last_change":"2026-02-18T06:44:45.723516+0000","last_active":"2026-02-18T06:44:45.723516+0000","last_peered":"2026-02-18T06:44:45.723516+0000","last_clean":"2026-02-18T06:44:45.723516+0000","last_became_active":"2026-02-18T06:20:08.541914+0000","last_became_peered":"2026-02-18T06:20:08.541914+0000","last_unstale":"2026-02-18T06:44:45.723516+0000","last_undegraded":"2026-02-18T06:44:45.723516+0000","last_fullsized":"2026-02-18T06:44:45.723516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:45.723474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:45.723474+0000","last_clean_scrub_stamp":"2026-02-18T06:44:45.723474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T06:56:42.909003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408427,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"183'3","reported_seq":377,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:43.976695+0000","last_change":"2026-02-18T06:44:43.976695+0000","last_active":"2026-02-18T06:44:43.976695+0000","last_peered":"2026-02-18T06:44:43.976695+0000","last_clean":"2026-02-18T06:44:43.976695+0000","last_became_active":"2026-02-18T06:20:08.541913+0000","last_became_peered":"2026-02-18T06:20:08.541913+0000","last_unstale":"2026-02-18T06:44:43.976695+0000","last_undegraded":"2026-02-18T06:44:43.976695+0000","last_fullsized":"2026-02-18T06:44:43.976695+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'3","last_scrub_stamp":"2026-02-18T06:44:43.976669+0000","last_deep_scrub":"183'3","last_deep_scrub_stamp":"2026-02-18T06:44:43.976669+0000","last_clean_scrub_stamp":"2026-02-18T06:44:43.976669+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T17:42:35.041773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077138100000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":375,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:44.738844+0000","last_change":"2026-02-18T06:44:44.738844+0000","last_active":"2026-02-18T06:44:44.738844+0000","last_peered":"2026-02-18T06:44:44.738844+0000","last_clean":"2026-02-18T06:44:44.738844+0000","last_became_active":"2026-02-18T06:20:08.541846+0000","last_became_peered":"2026-02-18T06:20:08.541846+0000","last_unstale":"2026-02-18T06:44:44.738844+0000","last_undegraded":"2026-02-18T06:44:44.738844+0000","last_fullsized":"2026-02-18T06:44:44.738844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:44.738735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:44.738735+0000","last_clean_scrub_stamp":"2026-02-18T06:44:44.738735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T13:25:42.338673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000219657,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":376,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:43.763801+0000","last_change":"2026-02-18T06:44:43.763801+0000","last_active":"2026-02-18T06:44:43.763801+0000","last_peered":"2026-02-18T06:44:43.763801+0000","last_clean":"2026-02-18T06:44:43.763801+0000","last_became_active":"2026-02-18T06:20:08.541956+0000","last_became_peered":"2026-02-18T06:20:08.541956+0000","last_unstale":"2026-02-18T06:44:43.763801+0000","last_undegraded":"2026-02-18T06:44:43.763801+0000","last_fullsized":"2026-02-18T06:44:43.763801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-18T06:44:43.763716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-18T06:44:43.763716+0000","last_clean_scrub_stamp":"2026-02-18T06:44:43.763716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T10:04:45.816747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023866299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"183'2","reported_seq":376,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:44.687790+0000","last_change":"2026-02-18T06:44:44.687790+0000","last_active":"2026-02-18T06:44:44.687790+0000","last_peered":"2026-02-18T06:44:44.687790+0000","last_clean":"2026-02-18T06:44:44.687790+0000","last_became_active":"2026-02-18T06:20:08.541386+0000","last_became_peered":"2026-02-18T06:20:08.541386+0000","last_unstale":"2026-02-18T06:44:44.687790+0000","last_undegraded":"2026-02-18T06:44:44.687790+0000","last_fullsized":"2026-02-18T06:44:44.687790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'2","last_scrub_stamp":"2026-02-18T06:44:44.687695+0000","last_deep_scrub":"183'2","last_deep_scrub_stamp":"2026-02-18T06:44:44.687695+0000","last_clean_scrub_stamp":"2026-02-18T06:44:44.687695+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T08:45:24.482567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00034991900000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"183'74","reported_seq":467,"reported_epoch":183,"state":"active+clean","last_fresh":"2026-02-18T06:44:43.733729+0000","last_change":"2026-02-18T06:44:43.733729+0000","last_active":"2026-02-18T06:44:43.733729+0000","last_peered":"2026-02-18T06:44:43.733729+0000","last_clean":"2026-02-18T06:44:43.733729+0000","last_became_active":"2026-02-18T06:20:05.693589+0000","last_became_peered":"2026-02-18T06:20:05.693589+0000","last_unstale":"2026-02-18T06:44:43.733729+0000","last_undegraded":"2026-02-18T06:44:43.733729+0000","last_fullsized":"2026-02-18T06:44:43.733729+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'74","last_scrub_stamp":"2026-02-18T06:44:43.733691+0000","last_deep_scrub":"183'74","last_deep_scrub_stamp":"2026-02-18T06:44:43.733691+0000","last_clean_scrub_stamp":"2026-02-18T06:44:43.733691+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T07:23:23.677214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136708,"kb_used_data":1524,"kb_used_omap":701,"kb_used_meta":134466,"kb_avail":94235132,"statfs":{"total":96636764160,"available":96496775168,"internally_reserved":0,"allocated":1560576,"data_stored":785639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718806,"internal_metadata":137693226},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738672,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":464,"kb_used_meta":134703,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475729,"internal_metadata":137936303},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137292,"kb_used_data":2108,"kb_used_omap":748,"kb_used_meta":134419,"kb_avail":94234548,"statfs":{"total":96636764160,"available":96496177152,"internally_reserved":0,"allocated":2158592,"data_stored":1376026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":766293,"internal_metadata":137645739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-18T06:45:04.021 DEBUG:teuthology.orchestra.run.trial031:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2026-02-18T06:45:04.212 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-18T06:45:04.212 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-18T06:45:04.212 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2026-02-18T06:45:04.212 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:45:04.257 INFO:tasks.ceph.cluster1.osd.0:Stopped 2026-02-18T06:45:04.257 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2026-02-18T06:45:04.257 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:45:04.305 INFO:tasks.ceph.cluster1.osd.1:Stopped 2026-02-18T06:45:04.305 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2026-02-18T06:45:04.305 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:45:04.346 INFO:tasks.ceph.cluster1.osd.2:Stopped 2026-02-18T06:45:04.346 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-18T06:45:04.346 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2026-02-18T06:45:04.346 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:45:04.417 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2026-02-18T06:45:04.417 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-18T06:45:04.417 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2026-02-18T06:45:04.418 INFO:teuthology.orchestra.run:waiting for 300 2026-02-18T06:45:04.467 INFO:tasks.ceph.cluster1.mon.a:Stopped 2026-02-18T06:45:04.468 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-18T06:45:04.468 DEBUG:teuthology.orchestra.run.trial031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-18T06:45:04.534 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:45:04.534 DEBUG:teuthology.orchestra.run.trial031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2026-02-18T06:45:04.608 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:45:04.608 DEBUG:teuthology.orchestra.run.trial031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2026-02-18T06:45:04.689 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:45:04.689 DEBUG:teuthology.orchestra.run.trial031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2026-02-18T06:45:04.770 INFO:tasks.ceph:Archiving mon data... 2026-02-18T06:45:04.771 DEBUG:teuthology.misc:Transferring archived files from trial031:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/data/cluster1.mon.a.tgz 2026-02-18T06:45:04.771 DEBUG:teuthology.orchestra.run.trial031:> mktemp 2026-02-18T06:45:04.827 INFO:teuthology.orchestra.run.trial031.stdout:/tmp/tmp.Pcrm30YhHi 2026-02-18T06:45:04.827 DEBUG:teuthology.orchestra.run.trial031:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.Pcrm30YhHi 2026-02-18T06:45:04.954 DEBUG:teuthology.orchestra.run.trial031:> sudo chmod 0666 /tmp/tmp.Pcrm30YhHi 2026-02-18T06:45:05.034 DEBUG:teuthology.orchestra.remote:trial031:/tmp/tmp.Pcrm30YhHi is 510KB 2026-02-18T06:45:05.130 DEBUG:teuthology.orchestra.run.trial031:> rm -fr /tmp/tmp.Pcrm30YhHi 2026-02-18T06:45:05.147 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-18T06:45:05.147 DEBUG:teuthology.orchestra.run.trial031:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-18T06:45:05.189 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-18T06:45:05.267 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-18T06:45:05.284 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-18T06:45:05.284 INFO:tasks.ceph:Archiving crash dumps... 2026-02-18T06:45:05.286 DEBUG:teuthology.misc:Transferring archived files from trial031:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/remote/trial031/crash 2026-02-18T06:45:05.288 DEBUG:teuthology.orchestra.run.trial031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-18T06:45:05.333 DEBUG:teuthology.misc:Transferring archived files from trial130:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/remote/trial130/crash 2026-02-18T06:45:05.334 DEBUG:teuthology.orchestra.run.trial130:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-18T06:45:05.357 INFO:tasks.ceph:Compressing logs... 2026-02-18T06:45:05.358 DEBUG:teuthology.orchestra.run.trial031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-18T06:45:05.378 DEBUG:teuthology.orchestra.run.trial130:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-18T06:45:05.403 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28394.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28345.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28394.log.gz 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28691.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28345.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28345.log.gz 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28691.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28691.log.gz 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29979.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29734.log 2026-02-18T06:45:05.404 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29979.log.gz 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28443.log 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29734.log.gz 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28443.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28443.log.gz 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25019.log 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28206.log 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.25019.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25019.log.gz 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29636.log 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28206.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28206.log.gz 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29587.log 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29636.log.gz 2026-02-18T06:45:05.405 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2026-02-18T06:45:05.406 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29587.log.gz 2026-02-18T06:45:05.406 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28692.log 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25520.log 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28546.log 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28692.log.gz 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.25520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28296.log 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr: 28.5% -- replaced with /var/log/ceph/cluster1-client.admin.25520.log.gz 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28546.log: gzip 0.0% -5 -- replaced with /var/log/ceph/cluster1-client.admin.28546.log.gz --verbose 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr: -- /var/log/ceph/cluster1-client.admin.28545.log 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25374.log 2026-02-18T06:45:05.415 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28296.log.gz 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28545.log.gz 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25200.log 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.25374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29489.log 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25374.log.gz 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.25200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25200.log.gz 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28835.log 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29489.log.gz 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29930.log 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29832.log 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28835.log.gz 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29783.log 2026-02-18T06:45:05.416 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29930.log.gz 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29832.log.gz 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28544.log 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29440.log 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29783.log.gz 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28544.log.gz 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28884.log 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29881.log 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29440.log.gz 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28884.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28884.log.gz 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28231.log 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29881.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29881.log.gz 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28693.log 2026-02-18T06:45:05.417 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28231.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.28231.log.gz 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28693.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28693.log.gz 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20021.log 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30026.log 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/ceph.tmp-client.admin.20021.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20021.log.gz 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28786.log 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.30026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25247.log 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30026.log.gz 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28786.log.gz 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24924.log 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.25247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29685.log 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25247.log.gz 2026-02-18T06:45:05.418 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.24924.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24924.log.gz 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28136.log 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29685.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29685.log.gz 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25153.log 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.28136.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28136.log.gz 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29538.log 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.25153.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25153.log.gz 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1-client.admin.29538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29538.log.gz 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr:/var/log/ceph/cluster1.log: 88.2% -- replaced with /var/log/ceph/cluster1.log.gz 2026-02-18T06:45:05.419 INFO:teuthology.orchestra.run.trial031.stderr: 93.8% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2026-02-18T06:45:05.422 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69855.log 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42078.log 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77333.log 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69855.log.gz 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.42078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61146.log 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42078.log.gz 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66798.log 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77333.log.gz 2026-02-18T06:45:05.431 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29444.log 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61146.log.gz 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73763.log 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29444.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29444.log.gz 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48670.log 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66798.log.gz 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57250.log 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73763.log.gz 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39514.log 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48670.log: /var/log/ceph/cluster1-client.admin.57250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43022.log 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.48670.log.gz 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39514.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.57250.log.gz 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47150.log 2026-02-18T06:45:05.432 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41932.log 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.43022.log.gz 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62989.log 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47150.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39514.log.gz 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76930.log 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41932.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.47150.log.gz 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51438.log 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.41932.log.gz 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76930.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62989.log.gz 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78094.log 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.51438.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.76930.log.gz 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28762.log 2026-02-18T06:45:05.433 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51438.log.gz 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.78094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59372.log 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70016.log 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28762.log.gz 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78094.log.gz 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44401.log 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59372.log.gz 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37547.log 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44401.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70016.log.gz 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75771.log 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.44401.log.gz 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.69411.log 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.37547.log: /var/log/ceph/cluster1-client.mirror.75771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.37228.log 2026-02-18T06:45:05.434 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.37547.log.gz 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75771.log.gz 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.69411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64233.log 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.2.37228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29395.log 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54716.log 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64233.log.gz 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68875.log 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29395.log.gz 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62968.log 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.54716.log.gz 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.68875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45035.log 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34803.log 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.68875.log.gz 2026-02-18T06:45:05.435 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49429.log 2026-02-18T06:45:05.436 INFO:teuthology.orchestra.run.trial130.stderr: 92.8% 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.69411.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.62968.log.gz 2026-02-18T06:45:05.436 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.436 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.45035.log.gz 2026-02-18T06:45:05.436 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.34803.log: /var/log/ceph/cluster1-client.admin.50867.log 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48699.log 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.34803.log.gz 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75832.log 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.49429.log.gz 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48699.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.50867.log.gz 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50610.log 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.48699.log.gz 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57959.log 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75832.log.gz 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51911.log 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.50610.log.gz 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.30897.log 2026-02-18T06:45:05.454 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.51911.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.57959.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76126.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51911.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.30897.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.30897.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59034.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.76126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47244.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.59034.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44806.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.59034.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70192.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76126.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.47244.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45732.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.44806.log: /var/log/ceph/cluster2-client.mirror.70192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60258.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.44806.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.33284.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60258.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45732.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37307.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.33284.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60258.log.gz 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57124.log 2026-02-18T06:45:05.455 INFO:teuthology.orchestra.run.trial130.stderr: 18.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.33284.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.37307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78614.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.57124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53808.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37307.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.78614.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57124.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71778.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78614.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52729.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.53808.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44051.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.52729.log: -- replaced with /var/log/ceph/cluster2-client.mirror.71778.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48150.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52729.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.44051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.33221.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36081.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.48150.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.44051.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.33221.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70192.log.gz 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71261.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71953.log 2026-02-18T06:45:05.456 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49033.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36081.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71261.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73471.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.49033.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61457.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.73471.log: gzip -5 --verbose 89.8% -- /var/log/ceph/cluster1-client.mirror.63172.log -- replaced with /var/log/ceph/cluster1-client.mirror.49033.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73471.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61457.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71953.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53896.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63172.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61457.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70225.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.53896.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63172.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35882.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70225.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53896.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64746.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70225.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45200.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75796.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35882.log.gz 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.45200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73402.log 2026-02-18T06:45:05.457 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45200.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80165.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75796.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78044.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.73402.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73402.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80165.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.78044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72069.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.37228.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.78044.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35114.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45791.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72069.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70780.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.45791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69793.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35114.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70780.log: gzip 90.1% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.45791.log.gz --verbose 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: -- /var/log/ceph/cluster2-client.mirror.40107.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65572.log 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: 87.8% -- replaced with /var/log/ceph/cluster2.log.gz 2026-02-18T06:45:05.458 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69793.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.40107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29346.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64746.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40107.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70780.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.65572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62122.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65572.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29346.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.admin.29346.log.gz /var/log/ceph/cluster2-client.mirror.59808.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47715.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68074.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47715.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59808.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49548.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.47715.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.68074.log: 84.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68074.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45413.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62122.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40170.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.49548.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56945.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66996.log 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.45413.log.gz 2026-02-18T06:45:05.459 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.40170.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.56945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35248.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66996.log: 90.0%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.56945.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.58655.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74833.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66996.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.58655.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.35248.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69831.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.58655.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49367.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.69831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65497.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69831.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52247.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74833.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.65497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58293.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.49367.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65497.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41125.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.52247.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.58293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66776.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:gzip/var/log/ceph/cluster2-client.mirror.41125.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.56199.log 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58293.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66776.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41125.log.gz 2026-02-18T06:45:05.460 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76786.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56199.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66776.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.37123.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.56199.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32915.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.0.37123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30896.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.76786.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.32915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51805.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.32915.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30896.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30896.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61960.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.51805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35310.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.33221.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.51805.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54348.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75461.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.35310.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.54348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75998.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63236.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75461.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54348.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61960.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75998.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75461.log.gz 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67106.log 2026-02-18T06:45:05.461 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75998.log.gz 2026-02-18T06:45:05.462 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71168.log 2026-02-18T06:45:05.462 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41285.log 2026-02-18T06:45:05.462 INFO:teuthology.orchestra.run.trial130.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63236.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70472.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67106.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71168.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.37167.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.41285.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30791.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70472.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.1.37167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37672.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44171.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.37672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58265.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.37672.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.44171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.73471.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.44171.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.58265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44975.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.73471.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.58265.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65595.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54160.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.44975.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.65595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60788.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65595.log.gz 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.54160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44230.log 2026-02-18T06:45:05.470 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54160.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53266.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.44230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72574.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60788.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44230.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.53266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48578.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72574.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53266.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47006.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.73471.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.48578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77657.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47006.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48578.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75305.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72574.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.47006.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70907.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77657.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75305.log: gzip -5 --verbose -- 86.8% /var/log/ceph/cluster1-client.admin.54409.log -- replaced with /var/log/ceph/cluster1-client.mirror.75305.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.70907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64515.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70907.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.54409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71414.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.54409.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75950.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.71414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75704.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76339.log 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75950.log.gz 2026-02-18T06:45:05.471 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75704.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64515.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30791.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71414.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76608.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75704.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66952.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76339.log: /var/log/ceph/cluster1-client.admin.76608.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76339.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57005.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.76608.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49579.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57005.log: gzip -5 --verbose -- 89.3% /var/log/ceph/cluster2-client.admin.29885.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66952.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49579.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.57005.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79251.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.49579.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29885.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29885.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35083.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.79251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32534.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77180.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32534.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.35083.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 71.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32534.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69729.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61682.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 83.2% -- replaced with /var/log/ceph/cluster1-client.mirror.77180.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36664.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69729.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36558.log 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61682.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36664.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36664.log.gz 2026-02-18T06:45:05.472 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46834.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36558.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36558.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30826.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.46834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36785.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71855.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30826.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46834.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63709.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36785.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70318.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79251.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71855.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63709.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63709.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31827.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39991.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70318.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.31827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53370.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.31827.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.39991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73124.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39991.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.37167.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64363.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.73124.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.53370.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36511.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.73124.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44336.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64363.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45851.log 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.44336.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36511.log.gz 2026-02-18T06:45:05.473 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50900.log 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44336.log.gz 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74379.log 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45851.log.gz 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39459.log 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.50900.log.gz 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.74379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57926.log 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.39459.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.74379.log.gz 2026-02-18T06:45:05.474 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41254.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.39459.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46368.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.57926.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62567.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46368.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.41254.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39253.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.46368.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80214.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39253.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62567.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60431.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39253.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80214.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80214.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70570.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67362.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60431.log: /var/log/ceph/cluster1-client.admin.70570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.73524.log 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.70570.log.gz 2026-02-18T06:45:05.477 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.78736.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67362.log: /var/log/ceph/cluster1-client.mirror.3.73524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76417.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.78736.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60431.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56051.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67362.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62530.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.76417.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78420.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.56051.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32721.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62530.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.78420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29787.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78420.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50491.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25473.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29787.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43963.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.73524.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.50491.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.25473.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.admin.48912.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.25473.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45066.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.43963.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48945.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45066.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.48912.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70507.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.45066.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30642.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.48945.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38989.log 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30642.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70507.log.gz 2026-02-18T06:45:05.478 INFO:teuthology.orchestra.run.trial130.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.78736.log.gz 2026-02-18T06:45:05.479 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67808.log 2026-02-18T06:45:05.479 INFO:teuthology.orchestra.run.trial130.stderr: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30642.log.gz 2026-02-18T06:45:05.479 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.38989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44368.log 2026-02-18T06:45:05.479 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.38989.log.gz 2026-02-18T06:45:05.479 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.67808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59212.log 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44368.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.67808.log.gz 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68707.log 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.44368.log.gz 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.59212.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32721.log.gz 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45265.log 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.59212.log.gz 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.68707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61057.log 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73367.log 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.45265.log.gz 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75196.log 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61057.log.gz 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68707.log.gz 2026-02-18T06:45:05.480 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.73367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60763.log 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39949.log 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75196.log.gz 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66930.log 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.39949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55092.log 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60763.log.gz 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39949.log.gz 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72149.log 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66930.log.gz 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62661.log 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72149.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.55092.log.gz 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25351.log 2026-02-18T06:45:05.481 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72149.log.gz 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36254.log 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.25351.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25351.log.gz 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69351.log 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62661.log.gz 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.36254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56139.log 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.36254.log.gz 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.69351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30659.log 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69351.log.gz 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61078.log 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.56139.log.gz 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30659.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30659.log.gz 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41901.log 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32346.log 2026-02-18T06:45:05.482 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41901.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61078.log.gz 2026-02-18T06:45:05.483 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62009.log 2026-02-18T06:45:05.483 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.41901.log.gz 2026-02-18T06:45:05.483 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39596.log 2026-02-18T06:45:05.483 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76033.log 2026-02-18T06:45:05.483 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57065.log 2026-02-18T06:45:05.483 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.39596.log.gz 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32346.log.gz 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61432.log 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76033.log.gz 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63515.log 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62009.log.gz 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61432.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.57065.log.gz 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76448.log 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61432.log.gz 2026-02-18T06:45:05.484 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41492.log 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45006.log 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63515.log.gz 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.41492.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.76448.log.gz 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59993.log 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45006.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41492.log.gz 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67250.log 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.45006.log.gz 2026-02-18T06:45:05.485 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71027.log 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59993.log.gz 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.67250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.77137.log 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67250.log.gz 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77780.log 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr: 71.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71027.log.gz 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.1.77137.log: 19.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.77137.log.gz 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35704.log 2026-02-18T06:45:05.486 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50670.log 2026-02-18T06:45:05.487 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45473.log 2026-02-18T06:45:05.487 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35704.log.gz 2026-02-18T06:45:05.487 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.50670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46655.log 2026-02-18T06:45:05.487 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45473.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.50670.log.gz 2026-02-18T06:45:05.488 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33914.log 2026-02-18T06:45:05.488 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.45473.log.gz 2026-02-18T06:45:05.488 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.46655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57036.log 2026-02-18T06:45:05.488 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.33914.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46655.log.gz 2026-02-18T06:45:05.488 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41838.log 2026-02-18T06:45:05.488 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57036.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.admin.33914.log.gz 2026-02-18T06:45:05.488 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46337.log 2026-02-18T06:45:05.489 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.57036.log.gz 2026-02-18T06:45:05.489 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31804.log 2026-02-18T06:45:05.489 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.41838.log.gz 2026-02-18T06:45:05.489 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71044.log 2026-02-18T06:45:05.489 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.46337.log.gz 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.31804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63342.log 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31804.log.gz 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71918.log 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71044.log.gz 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63279.log 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63342.log.gz 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.71918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69231.log 2026-02-18T06:45:05.490 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77780.log.gz 2026-02-18T06:45:05.491 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.71918.log.gz 2026-02-18T06:45:05.491 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70685.log 2026-02-18T06:45:05.491 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.69231.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63279.log.gz 2026-02-18T06:45:05.491 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42207.log 2026-02-18T06:45:05.491 INFO:teuthology.orchestra.run.trial130.stderr: 62.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69231.log.gz 2026-02-18T06:45:05.491 INFO:teuthology.orchestra.run.trial031.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2026-02-18T06:45:05.492 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.70685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74677.log 2026-02-18T06:45:05.492 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62593.log 2026-02-18T06:45:05.492 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74677.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.42207.log.gz 2026-02-18T06:45:05.492 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55611.log 2026-02-18T06:45:05.492 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74677.log.gz 2026-02-18T06:45:05.492 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62593.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70685.log.gz 2026-02-18T06:45:05.492 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55730.log 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55611.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62593.log.gz 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41192.log 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.55611.log.gz 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47374.log 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.55730.log.gz 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63797.log 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr: 88.6%/var/log/ceph/cluster2-client.mirror.47374.log: -- replaced with /var/log/ceph/cluster2-client.admin.41192.log.gz 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45942.log 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47374.log.gz 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31485.log 2026-02-18T06:45:05.493 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63797.log.gz 2026-02-18T06:45:05.494 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52491.log 2026-02-18T06:45:05.494 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31485.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.45942.log.gz 2026-02-18T06:45:05.494 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38030.log 2026-02-18T06:45:05.494 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31485.log.gz 2026-02-18T06:45:05.494 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51505.log 2026-02-18T06:45:05.494 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.52491.log.gz 2026-02-18T06:45:05.495 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41160.log 2026-02-18T06:45:05.495 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.51505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58136.log 2026-02-18T06:45:05.495 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38030.log.gz 2026-02-18T06:45:05.495 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.51505.log.gz 2026-02-18T06:45:05.495 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.41160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32483.log 2026-02-18T06:45:05.495 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41160.log.gz 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.58136.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32010.log 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58136.log.gz 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32790.log 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr: 65.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32483.log.gz 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77682.log 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57096.log 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32790.log.gz 2026-02-18T06:45:05.496 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77682.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77682.log.gz 2026-02-18T06:45:05.497 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31676.log 2026-02-18T06:45:05.497 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29934.log 2026-02-18T06:45:05.497 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57096.log: /var/log/ceph/cluster1-client.admin.31676.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.57096.log.gz 2026-02-18T06:45:05.497 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51876.log 2026-02-18T06:45:05.497 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.admin.31676.log.gz 2026-02-18T06:45:05.497 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29934.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29934.log.gz 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71135.log 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.51876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32213.log 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51876.log.gz 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31865.log 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31427.log 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32213.log.gz 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31865.log: 65.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31865.log.gz 2026-02-18T06:45:05.498 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43688.log 2026-02-18T06:45:05.499 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60203.log 2026-02-18T06:45:05.499 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31427.log.gz 2026-02-18T06:45:05.499 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30574.log 2026-02-18T06:45:05.499 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60203.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.43688.log.gz 2026-02-18T06:45:05.499 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59609.log 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial031.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.30574.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75496.log 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.30574.log.gz 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36639.log 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32010.log.gz 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60203.log.gz 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59609.log.gz 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42295.log 2026-02-18T06:45:05.500 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75496.log.gz 2026-02-18T06:45:05.501 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30032.log 2026-02-18T06:45:05.501 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36639.log.gz 2026-02-18T06:45:05.501 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76215.log 2026-02-18T06:45:05.501 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.42295.log.gz 2026-02-18T06:45:05.501 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.30032.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30032.log.gz 2026-02-18T06:45:05.501 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62218.log 2026-02-18T06:45:05.502 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76215.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71135.log.gz 2026-02-18T06:45:05.502 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32500.log 2026-02-18T06:45:05.502 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76215.log.gz 2026-02-18T06:45:05.502 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50988.log 2026-02-18T06:45:05.502 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70353.log 2026-02-18T06:45:05.502 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62218.log.gz 2026-02-18T06:45:05.502 INFO:teuthology.orchestra.run.trial130.stderr: 64.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32500.log.gz 2026-02-18T06:45:05.503 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.50988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63899.log 2026-02-18T06:45:05.503 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74410.log 2026-02-18T06:45:05.503 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70353.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50988.log.gz 2026-02-18T06:45:05.503 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62154.log 2026-02-18T06:45:05.503 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63899.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70353.log.gz 2026-02-18T06:45:05.503 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.74410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75080.log 2026-02-18T06:45:05.503 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63899.log.gz 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.74410.log.gz 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74522.log 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62154.log.gz 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.75080.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74522.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.75080.log.gz 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20067.log 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74522.log.gz 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5/var/log/ceph/cluster2.audit.log: --verbose -- /var/log/ceph/cluster1-client.admin.34564.log 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/ceph.tmp-client.admin.20067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38209.log 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20067.log.gz 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68135.log 2026-02-18T06:45:05.504 INFO:teuthology.orchestra.run.trial130.stderr:gzip/var/log/ceph/cluster1-client.mirror.38209.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.77182.log 2026-02-18T06:45:05.505 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.34564.log.gz 2026-02-18T06:45:05.505 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.68135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50800.log 2026-02-18T06:45:05.505 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68135.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.38209.log.gz 2026-02-18T06:45:05.505 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.505 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.2.77182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74559.log 2026-02-18T06:45:05.505 INFO:teuthology.orchestra.run.trial130.stderr: 19.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.77182.log.gz 2026-02-18T06:45:05.505 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.50800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32970.log 2026-02-18T06:45:05.506 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50800.log.gz 2026-02-18T06:45:05.506 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65321.log 2026-02-18T06:45:05.506 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74559.log.gz 2026-02-18T06:45:05.506 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64422.log 2026-02-18T06:45:05.506 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32970.log: /var/log/ceph/cluster2-client.mirror.65321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55891.log 2026-02-18T06:45:05.506 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65321.log.gz 2026-02-18T06:45:05.506 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73159.log 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64422.log.gz 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.55891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46029.log 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55891.log.gz 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32970.log.gz 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61589.log 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.73159.log: /var/log/ceph/cluster1-client.mirror.46029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32078.log 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46029.log.gz 2026-02-18T06:45:05.507 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55958.log 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77014.log 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61589.log.gz 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr: 93.7% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32078.log.gz 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76818.log 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73159.log.gz 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.55958.log.gz 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.77014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76708.log 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.77014.log.gz 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33622.log 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.76818.log.gz 2026-02-18T06:45:05.508 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63196.log 2026-02-18T06:45:05.509 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33622.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.76708.log.gz 2026-02-18T06:45:05.509 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60937.log 2026-02-18T06:45:05.509 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33622.log.gz 2026-02-18T06:45:05.509 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63055.log 2026-02-18T06:45:05.509 INFO:teuthology.orchestra.run.trial130.stderr: 63.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63196.log.gz 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56625.log 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60937.log.gz 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.68077.log 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63055.log.gz 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72686.log 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.56625.log.gz 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.68077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54866.log 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.68077.log.gz 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72607.log 2026-02-18T06:45:05.510 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72686.log.gz 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.54866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70539.log 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.54866.log.gz 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69248.log 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72607.log: gzip -5 --verbose/var/log/ceph/cluster1-client.admin.70539.log: -- /var/log/ceph/cluster1-client.admin.51686.log 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.72607.log.gz 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.69248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62418.log 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.70539.log.gz 2026-02-18T06:45:05.511 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.51686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47181.log 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.51686.log.gz 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.62418.log: -- 89.6% /var/log/ceph/cluster1-client.admin.46715.log 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69248.log.gz 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41404.log 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.47181.log.gz 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46715.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62418.log.gz 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35944.log 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.46715.log.gz 2026-02-18T06:45:05.512 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64670.log 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41404.log: /var/log/ceph/cluster1-client.admin.35944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60526.log 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.41404.log.gz 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.35944.log.gz 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55551.log 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64670.log.gz 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38653.log 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55551.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60526.log.gz 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78115.log 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.55551.log.gz 2026-02-18T06:45:05.513 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46596.log 2026-02-18T06:45:05.514 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38653.log.gz 2026-02-18T06:45:05.514 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30471.log 2026-02-18T06:45:05.514 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.78115.log: /var/log/ceph/cluster1-client.admin.46596.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.46596.log.gz 2026-02-18T06:45:05.514 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37019.log 2026-02-18T06:45:05.515 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70037.log 2026-02-18T06:45:05.515 INFO:teuthology.orchestra.run.trial130.stderr: 68.2% -- replaced with /var/log/ceph/cluster1-client.mirror.30471.log.gz 2026-02-18T06:45:05.515 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.37019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76577.log 2026-02-18T06:45:05.515 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37019.log.gz 2026-02-18T06:45:05.515 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65219.log 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr: 83.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70037.log.gz 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48841.log 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46158.log 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.65219.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.76577.log.gz 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65219.log.gz 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.48841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35279.log 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46158.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48841.log.gz 2026-02-18T06:45:05.516 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71328.log 2026-02-18T06:45:05.517 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.46158.log.gz 2026-02-18T06:45:05.517 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60865.log 2026-02-18T06:45:05.517 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35279.log: /var/log/ceph/cluster1-client.admin.71328.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35279.log.gz 2026-02-18T06:45:05.517 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44640.log 2026-02-18T06:45:05.517 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.71328.log.gz 2026-02-18T06:45:05.517 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62395.log 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60865.log.gz 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.44640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35548.log 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62395.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.44640.log.gz 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73790.log 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52429.log 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73790.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.35548.log.gz 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68749.log 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73790.log.gz 2026-02-18T06:45:05.518 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31848.log 2026-02-18T06:45:05.519 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.52429.log.gz 2026-02-18T06:45:05.519 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.68749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66864.log 2026-02-18T06:45:05.519 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.31848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71996.log 2026-02-18T06:45:05.519 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68749.log.gz 2026-02-18T06:45:05.519 INFO:teuthology.orchestra.run.trial130.stderr: 63.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31848.log.gz 2026-02-18T06:45:05.519 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56886.log 2026-02-18T06:45:05.519 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30270.log 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66864.log.gz 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56886.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71996.log.gz 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61190.log 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.56886.log.gz 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.30270.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/cluster1-client.admin.30270.log.gz /var/log/ceph/cluster1-client.mirror.69924.log 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34356.log 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61190.log.gz 2026-02-18T06:45:05.520 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78684.log 2026-02-18T06:45:05.521 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69924.log.gz 2026-02-18T06:45:05.521 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31885.log 2026-02-18T06:45:05.521 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62395.log.gz 2026-02-18T06:45:05.521 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.78684.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.34356.log.gz 2026-02-18T06:45:05.521 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38426.log 2026-02-18T06:45:05.521 INFO:teuthology.orchestra.run.trial130.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78684.log.gz 2026-02-18T06:45:05.521 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71710.log 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr: 71.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31885.log.gz 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.38426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34710.log 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.38426.log.gz 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30591.log 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75168.log 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.34710.log.gz 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.30591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47940.log 2026-02-18T06:45:05.522 INFO:teuthology.orchestra.run.trial130.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.30591.log.gz 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62703.log 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.47940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63259.log 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75168.log.gz 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47940.log.gz 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47775.log 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46905.log 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62703.log.gz 2026-02-18T06:45:05.523 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47775.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63259.log.gz 2026-02-18T06:45:05.524 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60160.log 2026-02-18T06:45:05.524 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.47775.log.gz 2026-02-18T06:45:05.524 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39856.log 2026-02-18T06:45:05.524 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.46905.log: /var/log/ceph/cluster1-client.mirror.60160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40723.log 2026-02-18T06:45:05.524 INFO:teuthology.orchestra.run.trial130.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46905.log.gz 2026-02-18T06:45:05.524 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.39856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50002.log 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39856.log.gz 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72531.log 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72127.log 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.40723.log.gz 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60160.log.gz 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72531.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.50002.log.gz 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63299.log 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62350.log 2026-02-18T06:45:05.525 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72127.log: /var/log/ceph/cluster2-client.mirror.63299.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72127.log.gz 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54289.log 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71710.log.gz 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72531.log.gz 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57891.log 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63299.log.gz 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62350.log.gz 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.54289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68409.log 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.54289.log.gz 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50429.log 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.68409.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.57891.log.gz 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62078.log 2026-02-18T06:45:05.526 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68409.log.gz 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.50429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37123.log 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62078.log: 88.6%gzip -- replaced with /var/log/ceph/cluster2-client.admin.50429.log.gz -5 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.63687.log 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.37123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42809.log 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35462.log 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63687.log.gz 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62078.log.gz 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.42809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42689.log 2026-02-18T06:45:05.527 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78534.log 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.42809.log.gz 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.42689.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.35462.log.gz 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47066.log 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.78534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71203.log 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42689.log.gz 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71602.log 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.78534.log.gz 2026-02-18T06:45:05.528 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.47066.log.gz 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52701.log 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71203.log.gz 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.71602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53998.log 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr: 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71602.log.gz 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44489.log 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.52701.log.gz 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48550.log 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr: 87.2%/var/log/ceph/cluster1-client.mirror.44489.log: -- replaced with /var/log/ceph/cluster1-client.admin.53998.log.gz 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35975.log 2026-02-18T06:45:05.529 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74039.log 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44489.log.gz 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.48550.log.gz 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28562.log 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33012.log 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.35975.log.gz 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74039.log.gz 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28562.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28562.log.gz 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73179.log 2026-02-18T06:45:05.530 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.33012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29591.log 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33012.log.gz 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70993.log 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30608.log 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29591.log.gz 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46627.log 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70993.log.gz 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.30608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75328.log 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr: 72.8% -- replaced with /var/log/ceph/cluster2-client.admin.30608.log.gz 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73179.log.gz 2026-02-18T06:45:05.531 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39176.log 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.46627.log.gz 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51138.log 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75328.log.gz 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50835.log 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39176.log: /var/log/ceph/cluster2-client.admin.51138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51473.log 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39176.log.gz 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.50835.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.51138.log.gz 2026-02-18T06:45:05.532 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31158.log 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50835.log.gz 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57312.log 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.51473.log: /var/log/ceph/cluster1-client.mirror.31158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61102.log 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr: 64.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31158.log.gz 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51473.log.gz 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43322.log 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.57312.log.gz 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76180.log 2026-02-18T06:45:05.533 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.43322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58750.log 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61102.log.gz 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38560.log 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43322.log.gz 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76180.log.gz 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.58750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63732.log 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58750.log.gz 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63597.log 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38560.log: /var/log/ceph/cluster1-client.mirror.63732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32439.log 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38560.log.gz 2026-02-18T06:45:05.534 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63732.log.gz 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.68135.log 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63597.log.gz 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47035.log 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32439.log.gz 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.68135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63820.log 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63773.log 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63820.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.47035.log.gz 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67383.log 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63820.log.gz 2026-02-18T06:45:05.535 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58201.log 2026-02-18T06:45:05.536 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76962.log 2026-02-18T06:45:05.536 INFO:teuthology.orchestra.run.trial130.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63773.log.gz 2026-02-18T06:45:05.536 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.58201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34169.log 2026-02-18T06:45:05.536 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.58201.log.gz 2026-02-18T06:45:05.536 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.77242.log 2026-02-18T06:45:05.536 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76962.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.68135.log.gz 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.34169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75282.log 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.admin.76962.log.gz 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67383.log.gz 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34169.log.gz 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.3.77242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54227.log 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr: 19.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.77242.log.gz 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58168.log 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75282.log.gz 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29297.log 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.58168.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.54227.log.gz 2026-02-18T06:45:05.537 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60411.log 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29297.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.58168.log.gz 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30625.log 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29297.log.gz 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59558.log 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49331.log 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr: 90.8% 62.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60411.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.30625.log.gz 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61500.log 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.49331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54587.log 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59558.log.gz 2026-02-18T06:45:05.538 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52522.log 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49331.log.gz 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.54587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54089.log 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54587.log.gz 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61500.log.gz 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61168.log 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.54089.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.52522.log.gz 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75611.log 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.37123.log.gz 2026-02-18T06:45:05.539 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61168.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54089.log.gz 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67407.log 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61168.log.gz 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43084.log 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75611.log.gz 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58622.log 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.78684.log 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67407.log.gz 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.33222.log 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.58622.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.43084.log.gz 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.78684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.73472.log 2026-02-18T06:45:05.540 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.58622.log.gz 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.33222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50397.log 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr: 18.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.33222.log.gz 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.73472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64342.log 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr: 19.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.73472.log.gz 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36589.log 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.50397.log: /var/log/ceph/cluster1-client.mirror.64342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46096.log 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50397.log.gz 2026-02-18T06:45:05.541 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64342.log.gz 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74760.log 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36589.log.gz 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62329.log 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.78684.log.gz 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48974.log 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.46096.log.gz 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74760.log.gz 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56477.log 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62329.log.gz 2026-02-18T06:45:05.542 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46467.log 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.48974.log.gz 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40464.log 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.56477.log.gz 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61003.log 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.46467.log: /var/log/ceph/cluster2-client.admin.40464.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46467.log.gz 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29738.log 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.40464.log.gz 2026-02-18T06:45:05.543 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44774.log 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61003.log.gz 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29738.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37123.log.gz 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36926.log 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29738.log.gz 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.44774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40933.log 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44774.log.gz 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25520.log 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36926.log.gz 2026-02-18T06:45:05.544 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67150.log 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40933.log: /var/log/ceph/cluster2-client.admin.25520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41464.log 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25520.log.gz 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.40933.log.gz 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.67150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56857.log 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.67150.log.gz 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31774.log 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.41464.log.gz 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38150.log 2026-02-18T06:45:05.545 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56857.log: /var/log/ceph/cluster2-client.mirror.31774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38008.log 2026-02-18T06:45:05.546 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.56857.log.gz 2026-02-18T06:45:05.546 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.38150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45532.log 2026-02-18T06:45:05.546 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.38008.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.38150.log.gz 2026-02-18T06:45:05.546 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49005.log 2026-02-18T06:45:05.546 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31774.log.gz 2026-02-18T06:45:05.546 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.45532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41598.log 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45532.log.gz 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41663.log 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38008.log.gz 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.49005.log.gz 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43569.log/var/log/ceph/cluster1-client.mirror.41598.log: 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41598.log.gz 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54835.log 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.41663.log.gz 2026-02-18T06:45:05.547 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79201.log 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43569.log: /var/log/ceph/cluster1-client.admin.54835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40313.log 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.43569.log.gz 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr: 88.2%/var/log/ceph/cluster2-client.admin.79201.log: -- replaced with /var/log/ceph/cluster1-client.admin.54835.log.gz 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32755.log 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.79201.log.gz 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.40313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51197.log 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.77180.log 2026-02-18T06:45:05.548 INFO:teuthology.orchestra.run.trial130.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40313.log.gz 2026-02-18T06:45:05.549 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32755.log.gz 2026-02-18T06:45:05.549 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.51197.log: /var/log/ceph/cluster1-client.admin.34201.log 2026-02-18T06:45:05.549 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.1.77180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39914.log 2026-02-18T06:45:05.549 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51197.log.gz 2026-02-18T06:45:05.549 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54654.log 2026-02-18T06:45:05.549 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.39914.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34201.log.gz 2026-02-18T06:45:05.549 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2026-02-18T06:45:05.550 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39914.log.gz 2026-02-18T06:45:05.550 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.54654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.33178.log 2026-02-18T06:45:05.550 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50701.log 2026-02-18T06:45:05.550 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.54654.log.gz 2026-02-18T06:45:05.550 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.33178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62101.log 2026-02-18T06:45:05.551 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.50701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74650.log 2026-02-18T06:45:05.551 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.50701.log.gz 2026-02-18T06:45:05.551 INFO:teuthology.orchestra.run.trial130.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.77180.log.gz 2026-02-18T06:45:05.551 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53518.log 2026-02-18T06:45:05.551 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28910.log 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74650.log.gz 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53518.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62101.log.gz 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30693.log 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28910.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.53518.log.gz 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28910.log.gz 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31092.log 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30693.log: gzip -5 --verbose 68.1% -- -- replaced with /var/log/ceph/cluster2-client.mirror.30693.log.gz /var/log/ceph/cluster2-client.mirror.47655.log 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31092.log: 65.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31092.log.gz 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60505.log 2026-02-18T06:45:05.552 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.47655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58688.log 2026-02-18T06:45:05.553 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43055.log 2026-02-18T06:45:05.553 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47655.log.gz 2026-02-18T06:45:05.553 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.58688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70078.log 2026-02-18T06:45:05.553 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60505.log.gz 2026-02-18T06:45:05.553 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.58688.log.gz 2026-02-18T06:45:05.553 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57550.log 2026-02-18T06:45:05.553 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.43055.log.gz 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42497.log 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70078.log.gz 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.57550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61027.log 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.42497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79461.log 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57550.log.gz 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42497.log.gz 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54775.log 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.79461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46189.log 2026-02-18T06:45:05.554 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79461.log.gz 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.54775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32311.log 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61027.log.gz 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74495.log 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54775.log.gz 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.32311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68639.log 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.46189.log.gz 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.32311.log.gz 2026-02-18T06:45:05.555 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60103.log 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74495.log.gz 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.68639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72016.log 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.60103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31645.log 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.68639.log.gz 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67671.log 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.31645.log: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.60103.log.gz 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77074.log 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.31645.log.gz 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63078.log 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67671.log.gz 2026-02-18T06:45:05.556 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48181.log 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63078.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72016.log.gz 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77074.log.gz 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36223.log 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63078.log.gz 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54195.log 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.48181.log.gz 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61740.log 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.36223.log: /var/log/ceph/cluster2-client.mirror.54195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34595.log 2026-02-18T06:45:05.557 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.36223.log.gz 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54195.log.gz 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40026.log 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61740.log.gz 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37988.log 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.40026.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34595.log.gz 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65453.log 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40026.log.gz 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.37988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40199.log 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.65453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.37168.log 2026-02-18T06:45:05.558 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65453.log.gz 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33145.log 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.2.37168.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.40199.log.gz 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54954.log 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.37168.log.gz 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37988.log.gz 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37860.log 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.54954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.30792.log 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.37860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52550.log 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54954.log.gz 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.37860.log.gz 2026-02-18T06:45:05.559 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.30792.log: 18.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.30792.log.gz 2026-02-18T06:45:05.560 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39020.log 2026-02-18T06:45:05.560 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53549.log 2026-02-18T06:45:05.560 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.52550.log: /var/log/ceph/cluster2-client.admin.39020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30294.log 2026-02-18T06:45:05.560 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.78580.log 2026-02-18T06:45:05.560 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.39020.log.gz 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52550.log.gz 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.53549.log.gz 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.30294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57462.log 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30294.log.gz 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.78580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38275.log 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.78580.log.gz 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43992.log 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.57462.log.gz 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41630.log 2026-02-18T06:45:05.561 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.38275.log: /var/log/ceph/cluster1-client.admin.43992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60385.log 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41630.log: gzip 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.43992.log.gz -5 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.30710.log 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38275.log.gz 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60385.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.41630.log.gz 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60550.log 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33145.log.gz 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40839.log 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr: 64.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30710.log.gz 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60385.log.gz 2026-02-18T06:45:05.562 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42840.log 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57656.log 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60550.log.gz/var/log/ceph/cluster2-client.admin.42840.log: 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40839.log.gz 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36436.log 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.42840.log.gz 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.57656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.30826.log 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57656.log.gz 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40663.log 2026-02-18T06:45:05.563 INFO:teuthology.orchestra.run.trial130.stderr: 80.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36436.log.gz 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.30826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63444.log 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.30972.log 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.40663.log.gz 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79274.log 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63444.log.gz 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.30972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72732.log 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.79274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49488.log 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72732.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72732.log.gz 2026-02-18T06:45:05.564 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 89.3% --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.79274.log.gz -- 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr: /var/log/ceph/cluster2-client.admin.80067.log 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41751.log 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.49488.log: /var/log/ceph/cluster2-client.admin.80067.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80067.log.gz 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42326.log 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.30826.log.gz 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.41751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63542.log 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49488.log.gz 2026-02-18T06:45:05.565 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58018.log 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41751.log.gz 87.1% 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.42326.log.gz 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40902.log 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63542.log.gz 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32992.log 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.58018.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.30972.log.gz 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45325.log 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58018.log.gz 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44839.log 2026-02-18T06:45:05.566 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.40902.log.gz 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32992.log.gz 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64036.log 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45325.log: /var/log/ceph/cluster2-client.admin.44839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71568.log 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.45325.log.gz 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.44839.log.gz 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46775.log 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64036.log.gz 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71568.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71568.log.gz 2026-02-18T06:45:05.567 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35217.log 2026-02-18T06:45:05.568 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74200.log 2026-02-18T06:45:05.568 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35217.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.46775.log.gz 2026-02-18T06:45:05.568 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31731.log 2026-02-18T06:45:05.568 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35217.log.gz 2026-02-18T06:45:05.568 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51396.log 2026-02-18T06:45:05.568 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74200.log.gz 2026-02-18T06:45:05.568 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.31731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64187.log 2026-02-18T06:45:05.569 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.51396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79822.log 2026-02-18T06:45:05.569 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51396.log.gz 2026-02-18T06:45:05.569 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61923.log 2026-02-18T06:45:05.569 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.31731.log.gz 2026-02-18T06:45:05.569 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.79822.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79822.log.gz 2026-02-18T06:45:05.569 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67613.log 2026-02-18T06:45:05.569 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.61923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74059.log 2026-02-18T06:45:05.570 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64187.log.gz 2026-02-18T06:45:05.570 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53837.log 2026-02-18T06:45:05.570 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67613.log.gz 2026-02-18T06:45:05.570 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.61923.log.gz 2026-02-18T06:45:05.570 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39821.log 2026-02-18T06:45:05.570 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53399.log 2026-02-18T06:45:05.570 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.53837.log.gz 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.39821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77045.log 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39821.log.gz 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.69307.log 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74059.log.gz 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.53399.log.gz 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.77045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79334.log 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.77045.log.gz 2026-02-18T06:45:05.571 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.69307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42147.log 2026-02-18T06:45:05.572 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.79334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76273.log 2026-02-18T06:45:05.572 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.42147.log: gzip 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79334.log.gz -5 2026-02-18T06:45:05.572 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.53868.log 2026-02-18T06:45:05.572 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.42147.log.gz 2026-02-18T06:45:05.572 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32609.log 2026-02-18T06:45:05.572 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76273.log.gz 2026-02-18T06:45:05.572 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70734.log 2026-02-18T06:45:05.573 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.53868.log.gz 2026-02-18T06:45:05.573 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30522.log 2026-02-18T06:45:05.573 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32609.log.gz 2026-02-18T06:45:05.573 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.70734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62947.log 2026-02-18T06:45:05.573 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.30522.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30522.log.gz 2026-02-18T06:45:05.573 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56317.log 2026-02-18T06:45:05.574 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53095.log 2026-02-18T06:45:05.574 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70734.log.gz 2026-02-18T06:45:05.574 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.56317.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62947.log.gz 2026-02-18T06:45:05.574 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75351.log 2026-02-18T06:45:05.574 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56317.log.gz 2026-02-18T06:45:05.574 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30318.log 2026-02-18T06:45:05.574 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.53095.log.gz 2026-02-18T06:45:05.575 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56352.log 2026-02-18T06:45:05.575 INFO:teuthology.orchestra.run.trial130.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75351.log.gz 2026-02-18T06:45:05.575 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.30318.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.30318.log.gz 2026-02-18T06:45:05.575 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33793.log 2026-02-18T06:45:05.575 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.56352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59494.log 2026-02-18T06:45:05.575 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56352.log.gz 2026-02-18T06:45:05.575 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.33793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49750.log 2026-02-18T06:45:05.576 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.33793.log.gz 2026-02-18T06:45:05.576 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75854.log 2026-02-18T06:45:05.576 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.49750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31209.log 2026-02-18T06:45:05.576 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49750.log.gz 2026-02-18T06:45:05.576 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75854.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75854.log.gz 2026-02-18T06:45:05.576 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61322.log 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32517.log 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr: 64.6% -- replaced with /var/log/ceph/cluster2-client.mirror.31209.log.gz 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54260.log 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61322.log.gz 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32517.log: 71.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32517.log.gz 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45763.log 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61248.log 2026-02-18T06:45:05.577 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.54260.log.gz 2026-02-18T06:45:05.578 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57750.log 2026-02-18T06:45:05.578 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.45763.log.gz 2026-02-18T06:45:05.578 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48457.log 2026-02-18T06:45:05.578 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61248.log.gz 2026-02-18T06:45:05.578 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38395.log 2026-02-18T06:45:05.578 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.57750.log.gz 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39141.log 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.48457.log.gz 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.38395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78736.log 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.38395.log.gz 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72709.log 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39141.log.gz 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.78736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76308.log 2026-02-18T06:45:05.579 INFO:teuthology.orchestra.run.trial130.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59494.log.gz 2026-02-18T06:45:05.580 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38716.log 2026-02-18T06:45:05.580 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72709.log.gz 2026-02-18T06:45:05.580 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35493.log 2026-02-18T06:45:05.580 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76308.log.gz 2026-02-18T06:45:05.580 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60960.log 2026-02-18T06:45:05.580 INFO:teuthology.orchestra.run.trial130.stderr: 87.0%/var/log/ceph/cluster1-client.admin.35493.log: -- replaced with /var/log/ceph/cluster1-client.admin.38716.log.gz 2026-02-18T06:45:05.580 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41433.log 2026-02-18T06:45:05.581 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60960.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.35493.log.gz 2026-02-18T06:45:05.581 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2026-02-18T06:45:05.581 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60960.log.gz 2026-02-18T06:45:05.581 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58353.log 2026-02-18T06:45:05.581 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.41433.log.gz 2026-02-18T06:45:05.581 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59340.log 2026-02-18T06:45:05.581 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.58353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57688.log 2026-02-18T06:45:05.582 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.58353.log.gz 2026-02-18T06:45:05.582 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.59340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57343.log 2026-02-18T06:45:05.582 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79168.log 2026-02-18T06:45:05.582 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34937.log 88.5% 2026-02-18T06:45:05.582 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.57688.log.gz 2026-02-18T06:45:05.582 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.57343.log.gz 2026-02-18T06:45:05.582 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.79168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.69412.log 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59340.log.gz 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34937.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.admin.79168.log.gz 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56230.log 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.69412.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.34937.log.gz 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.69412.log.gz 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39715.log 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74270.log 2026-02-18T06:45:05.583 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.56230.log.gz 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.39715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72227.log 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74270.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39715.log.gz 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50227.log 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74270.log.gz 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58897.log 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72227.log.gz 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.50227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49460.log 2026-02-18T06:45:05.584 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50227.log.gz 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.58897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58384.log 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49460.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58897.log.gz 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78736.log.gz 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38685.log 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.49460.log.gz 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.58384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25876.log 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.58384.log.gz 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34041.log 2026-02-18T06:45:05.585 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.25876.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.38685.log.gz 2026-02-18T06:45:05.586 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38454.log 2026-02-18T06:45:05.586 INFO:teuthology.orchestra.run.trial130.stderr: 27.9% -- replaced with /var/log/ceph/cluster2-client.admin.25876.log.gz 2026-02-18T06:45:05.586 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.34041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70714.log 2026-02-18T06:45:05.586 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34041.log.gz 2026-02-18T06:45:05.586 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36689.log 2026-02-18T06:45:05.586 INFO:teuthology.orchestra.run.trial130.stderr: 89.2%/var/log/ceph/cluster2-client.mirror.70714.log: -- replaced with /var/log/ceph/cluster2-client.mirror.38454.log.gz 2026-02-18T06:45:05.586 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.73402.log 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36689.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36689.log.gz 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68575.log 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.73402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46806.log 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70714.log.gz 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.68575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70430.log 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.68575.log.gz 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74853.log 2026-02-18T06:45:05.587 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.46806.log.gz 2026-02-18T06:45:05.588 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42043.log 2026-02-18T06:45:05.588 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70430.log.gz 2026-02-18T06:45:05.588 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33741.log 2026-02-18T06:45:05.588 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.42043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54061.log 2026-02-18T06:45:05.588 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42043.log.gz 2026-02-18T06:45:05.588 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.33741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77607.log 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78115.log.gz 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.33741.log.gz 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74853.log.gz 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47567.log 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.73402.log.gz 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.54061.log.gz 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77607.log: gzip 0.0% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.77607.log.gz --verbose 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr: -- /var/log/ceph/cluster1-client.mirror.52812.log 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64209.log 2026-02-18T06:45:05.589 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.47567.log.gz 2026-02-18T06:45:05.590 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.52812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42386.log 2026-02-18T06:45:05.590 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52812.log.gz 2026-02-18T06:45:05.590 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49667.log 2026-02-18T06:45:05.590 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.42386.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64209.log.gz 2026-02-18T06:45:05.590 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51104.log 2026-02-18T06:45:05.590 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.42386.log.gz 2026-02-18T06:45:05.590 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.49667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63117.log 2026-02-18T06:45:05.591 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49667.log.gz/var/log/ceph/cluster2-client.admin.51104.log: 2026-02-18T06:45:05.591 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60577.log 2026-02-18T06:45:05.591 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.51104.log.gz 2026-02-18T06:45:05.591 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63117.log: gzip 86.9% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.63117.log.gz --verbose 2026-02-18T06:45:05.591 INFO:teuthology.orchestra.run.trial130.stderr: -- /var/log/ceph/cluster2-client.mirror.38525.log 2026-02-18T06:45:05.591 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34772.log 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65542.log 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60577.log.gz 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38525.log.gz 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70862.log 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.34772.log.gz 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.65542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59537.log 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.70862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32588.log 2026-02-18T06:45:05.592 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70862.log.gz 2026-02-18T06:45:05.593 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39428.log 2026-02-18T06:45:05.593 INFO:teuthology.orchestra.run.trial130.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65542.log.gz 2026-02-18T06:45:05.593 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32588.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59537.log.gz 2026-02-18T06:45:05.593 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.32588.log.gz 2026-02-18T06:45:05.593 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54528.log 2026-02-18T06:45:05.593 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40072.log 2026-02-18T06:45:05.593 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.39428.log: /var/log/ceph/cluster1-client.mirror.54528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40751.log 2026-02-18T06:45:05.594 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.39428.log.gz 2026-02-18T06:45:05.594 INFO:teuthology.orchestra.run.trial130.stderr: 89.1%/var/log/ceph/cluster2-client.mirror.40072.log: -- replaced with /var/log/ceph/cluster1-client.mirror.54528.log.gz 2026-02-18T06:45:05.594 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59245.log 2026-02-18T06:45:05.594 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40072.log.gz 2026-02-18T06:45:05.594 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.40751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77706.log 2026-02-18T06:45:05.594 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.59245.log: gzip 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40751.log.gz -5 2026-02-18T06:45:05.594 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.28660.log 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.59245.log.gz 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.77706.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77706.log.gz 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71296.log 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28660.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28660.log.gz 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43440.log 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75260.log 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71296.log.gz 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.43440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75143.log 2026-02-18T06:45:05.595 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43440.log.gz 2026-02-18T06:45:05.596 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69411.log 2026-02-18T06:45:05.596 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75260.log.gz 2026-02-18T06:45:05.596 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60475.log 2026-02-18T06:45:05.596 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75143.log.gz 2026-02-18T06:45:05.596 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.69411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68460.log 2026-02-18T06:45:05.596 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69411.log.gz 2026-02-18T06:45:05.596 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30744.log 2026-02-18T06:45:05.597 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.68460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46567.log 2026-02-18T06:45:05.597 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.68460.log.gz 2026-02-18T06:45:05.597 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30744.log: 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30744.log.gz 2026-02-18T06:45:05.597 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59875.log 2026-02-18T06:45:05.597 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46567.log: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60475.log.gz 2026-02-18T06:45:05.597 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72252.log 2026-02-18T06:45:05.597 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.46567.log.gz 2026-02-18T06:45:05.598 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.59875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69307.log 2026-02-18T06:45:05.598 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59875.log.gz 2026-02-18T06:45:05.598 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71513.log 2026-02-18T06:45:05.598 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.69307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44023.log 2026-02-18T06:45:05.598 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.71513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75579.log 2026-02-18T06:45:05.598 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71513.log.gz 2026-02-18T06:45:05.599 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52180.log 2026-02-18T06:45:05.599 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.44023.log.gz 2026-02-18T06:45:05.599 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52005.log 2026-02-18T06:45:05.599 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.69480.log 2026-02-18T06:45:05.599 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75579.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.52180.log.gz 2026-02-18T06:45:05.599 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.599 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36133.log 2026-02-18T06:45:05.600 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.52005.log.gz 2026-02-18T06:45:05.600 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.69480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71820.log 2026-02-18T06:45:05.600 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.36133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73306.log 2026-02-18T06:45:05.600 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.36133.log.gz 2026-02-18T06:45:05.600 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51169.log 2026-02-18T06:45:05.600 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71820.log.gz 2026-02-18T06:45:05.600 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30973.log 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73306.log.gz 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.51169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43859.log 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30973.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.51169.log.gz 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr: 65.1% -- replaced with /var/log/ceph/cluster1-client.mirror.30973.log.gz 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61521.log 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72252.log.gz 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.43859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44668.log 2026-02-18T06:45:05.601 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43859.log.gz 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67183.log 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.44668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43717.log 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.67183.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44668.log.gz 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59275.log 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.3.69480.log.gz 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67183.log.gz 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46973.log 2026-02-18T06:45:05.602 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.43717.log.gz 2026-02-18T06:45:05.603 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.59275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30972.log 2026-02-18T06:45:05.603 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71534.log 2026-02-18T06:45:05.603 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.46973.log.gz 2026-02-18T06:45:05.603 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30972.log: gzip 87.4% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.30972.log.gz --verbose 2026-02-18T06:45:05.603 INFO:teuthology.orchestra.run.trial130.stderr: -- /var/log/ceph/cluster1-client.admin.50033.log 2026-02-18T06:45:05.603 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.71534.log: 55.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71534.log.gz 2026-02-18T06:45:05.603 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.37305.log 2026-02-18T06:45:05.604 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56596.log 2026-02-18T06:45:05.604 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.50033.log.gz 2026-02-18T06:45:05.604 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.3.37305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32280.log 2026-02-18T06:45:05.604 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62058.log 2026-02-18T06:45:05.604 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.56596.log.gz 2026-02-18T06:45:05.605 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.32280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77241.log 2026-02-18T06:45:05.605 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.32280.log.gz 2026-02-18T06:45:05.605 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48386.log 2026-02-18T06:45:05.605 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77241.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61521.log.gz 2026-02-18T06:45:05.605 INFO:teuthology.orchestra.run.trial130.stderr: 80.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77241.log.gz 2026-02-18T06:45:05.605 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54747.log 2026-02-18T06:45:05.605 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.48386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29150.log 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48386.log.gz 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62058.log.gz 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57218.log 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.54747.log.gz 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59275.log.gz 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29150.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29150.log.gz 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68494.log 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.57218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63954.log 2026-02-18T06:45:05.606 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57218.log.gz 2026-02-18T06:45:05.607 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.68494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.78685.log 2026-02-18T06:45:05.607 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.68494.log.gz 2026-02-18T06:45:05.607 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59763.log 2026-02-18T06:45:05.607 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63954.log.gz 2026-02-18T06:45:05.607 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.78685.log: 19.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.78685.log.gz 2026-02-18T06:45:05.607 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48275.log 2026-02-18T06:45:05.607 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54687.log 2026-02-18T06:45:05.608 INFO:teuthology.orchestra.run.trial130.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59763.log.gz 2026-02-18T06:45:05.608 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32630.log 2026-02-18T06:45:05.608 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.48275.log.gz 2026-02-18T06:45:05.608 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.54687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53743.log 2026-02-18T06:45:05.608 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.54687.log.gz 2026-02-18T06:45:05.608 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52976.log 2026-02-18T06:45:05.609 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.53743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55518.log 2026-02-18T06:45:05.609 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32630.log.gz 2026-02-18T06:45:05.609 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53743.log.gz 2026-02-18T06:45:05.609 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44549.log 2026-02-18T06:45:05.609 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.52976.log.gz 2026-02-18T06:45:05.609 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39627.log 2026-02-18T06:45:05.609 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35767.log 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.55518.log.gz 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.44549.log.gz 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.39627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68197.log 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39627.log.gz 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37953.log 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35767.log.gz 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.68197.log: gzip -5 --verbose 88.6% -- -- replaced with /var/log/ceph/cluster2-client.mirror.68197.log.gz /var/log/ceph/cluster1-client.mirror.39330.log 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.610 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.37953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62832.log 2026-02-18T06:45:05.611 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.37953.log.gz 2026-02-18T06:45:05.611 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64001.log 2026-02-18T06:45:05.611 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39330.log.gz 2026-02-18T06:45:05.611 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60140.log 2026-02-18T06:45:05.611 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62832.log.gz 2026-02-18T06:45:05.611 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36614.log 2026-02-18T06:45:05.611 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64001.log.gz 2026-02-18T06:45:05.612 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29836.log 2026-02-18T06:45:05.612 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36614.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36614.log.gz 2026-02-18T06:45:05.612 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40139.log 2026-02-18T06:45:05.612 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29836.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29836.log.gz 2026-02-18T06:45:05.612 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70260.log 2026-02-18T06:45:05.612 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63925.log 2026-02-18T06:45:05.612 INFO:teuthology.orchestra.run.trial130.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60140.log.gz 2026-02-18T06:45:05.613 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.40139.log.gz 2026-02-18T06:45:05.613 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79295.log 2026-02-18T06:45:05.613 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70260.log.gz 2026-02-18T06:45:05.613 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42414.log 2026-02-18T06:45:05.613 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63925.log.gz 2026-02-18T06:45:05.613 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.79295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65522.log 2026-02-18T06:45:05.613 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.42414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58590.log 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.65522.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42414.log.gz 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62507.log 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65522.log.gz 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.58590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64564.log 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58590.log.gz 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.68039.log 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.69307.log.gz 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62507.log.gz 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.64564.log: gzip -5 --verbose 80.1% -- /var/log/ceph/cluster2-client.mirror.60282.log -- replaced with /var/log/ceph/cluster1-client.admin.64564.log.gz 2026-02-18T06:45:05.614 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.615 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.68039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2026-02-18T06:45:05.615 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67084.log 2026-02-18T06:45:05.615 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-mon.a.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60282.log.gz 2026-02-18T06:45:05.615 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48005.log 2026-02-18T06:45:05.615 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43234.log 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66842.log 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67084.log.gz 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.48005.log.gz 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43234.log: 96.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79295.log.gz 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57431.log 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69307.log.gz 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.43234.log.gz 2026-02-18T06:45:05.616 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55273.log 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57865.log 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66842.log.gz 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.57431.log.gz 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70170.log 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57865.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.55273.log.gz 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48210.log 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.57865.log.gz 2026-02-18T06:45:05.617 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34134.log 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37305.log 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.34134.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.48210.log.gz 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53124.log 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.37305.log.gz 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34134.log.gz 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.37305.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34076.log 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37305.log.gz 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62295.log 2026-02-18T06:45:05.618 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.34076.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.53124.log.gz 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42110.log 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.34076.log.gz 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65419.log 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62295.log.gz 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.42110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49400.log 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.65419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47275.log 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.42110.log.gz 2026-02-18T06:45:05.619 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74627.log 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.49400.log.gz 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.65419.log.gz 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31919.log 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.47275.log.gz 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62460.log 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74627.log.gz 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46064.log 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.31919.log.gz 2026-02-18T06:45:05.620 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50462.log 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.68039.log.gz 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.46064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55125.log 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70170.log.gz 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46064.log.gz 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.50462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56170.log 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.50462.log.gz 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25265.log 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.55125.log.gz 2026-02-18T06:45:05.621 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73839.log 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.56170.log.gz 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.25265.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25265.log.gz -5 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.0.73358.log 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34875.log 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73839.log.gz 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.73358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71973.log 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.73358.log.gz 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54030.log 2026-02-18T06:45:05.622 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.34875.log.gz 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.71973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49854.log 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr: 95.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73367.log.gz 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37922.log 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49854.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54030.log.gz 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49821.log 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.49854.log.gz 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71973.log.gz 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.37922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37760.log 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49821.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.37922.log.gz 2026-02-18T06:45:05.623 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69878.log 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.37760.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.49821.log.gz 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72759.log 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62460.log.gz 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59470.log 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69878.log.gz 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49124.log 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59470.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.72759.log.gzgzip 2026-02-18T06:45:05.624 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.43930.log 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59470.log.gz 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63401.log 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43930.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.49124.log.gz 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49942.log 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63401.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.43930.log.gz 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48065.log 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74795.log 2026-02-18T06:45:05.625 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.49942.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63401.log.gz 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28761.log 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.48065.log.gz 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53957.log 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr: 90.0%/var/log/ceph/cluster2-client.mirror.74795.log: -- replaced with /var/log/ceph/cluster1-client.mirror.49942.log.gz 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28761.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74795.log.gz 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28761.log.gz 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55451.log 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.33169.log 2026-02-18T06:45:05.626 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.55451.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.53957.log.gz 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45638.log 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55451.log.gz 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.33169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34387.log 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr: 18.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.33169.log.gz 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37760.log.gz 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.45638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48093.log 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45638.log.gz 2026-02-18T06:45:05.627 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76988.log 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.48093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72378.log 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.34387.log.gz 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52275.log 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.admin.76988.log.gz 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48093.log.gz 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56917.log 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72378.log.gz 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.52275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64457.log 2026-02-18T06:45:05.628 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72445.log 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52275.log.gz 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64457.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.56917.log.gz 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47627.log 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64457.log.gz 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.72445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32462.log 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47627.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.72445.log.gz 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68910.log 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32462.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.47627.log.gz 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75739.log 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.32462.log.gz 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.68910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47834.log 2026-02-18T06:45:05.629 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75739.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.68910.log.gz 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47409.log 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75739.log.gz 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.47834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77437.log 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.47409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74442.log 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47834.log.gz 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47409.log.gz 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68002.log 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77437.log.gz 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51567.log 2026-02-18T06:45:05.630 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74442.log.gz 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.68002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40492.log 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.51567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41867.log 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.40492.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68002.log.gz 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66732.log 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.51567.log.gz 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36049.log 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40492.log.gz 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.41867.log.gz 2026-02-18T06:45:05.631 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76529.log 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41563.log 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66732.log.gz 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38090.log 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76529.log.gz 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.41563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69901.log 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41563.log.gz 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70056.log 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38090.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36049.log.gz 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70929.log 2026-02-18T06:45:05.632 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.38090.log.gz 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69901.log.gz 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55185.log 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70056.log.gz 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29066.log 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70929.log.gz 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51361.log 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29066.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.55185.log.gz 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62175.log 2026-02-18T06:45:05.633 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29066.log.gz 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.51361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55154.log 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51361.log.gz 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51943.log 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55333.log 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.51943.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.55154.log.gz 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44202.log 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55333.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.51943.log.gz 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76903.log 2026-02-18T06:45:05.634 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.55333.log.gz 2026-02-18T06:45:05.635 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.44202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77136.log 2026-02-18T06:45:05.635 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.44202.log.gz 2026-02-18T06:45:05.635 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.76903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50362.log 2026-02-18T06:45:05.635 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79437.log 2026-02-18T06:45:05.635 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.76903.log.gz 2026-02-18T06:45:05.635 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.50362.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77136.log.gz 2026-02-18T06:45:05.635 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47972.log 2026-02-18T06:45:05.636 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50362.log.gz 2026-02-18T06:45:05.636 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.79437.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79437.log.gz 2026-02-18T06:45:05.636 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66908.log 2026-02-18T06:45:05.636 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30676.log 2026-02-18T06:45:05.636 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.47972.log.gz 2026-02-18T06:45:05.636 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62617.log 2026-02-18T06:45:05.636 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66908.log: /var/log/ceph/cluster2-client.mirror.30676.log: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30676.log.gz 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44739.log 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66908.log.gz 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42951.log 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62617.log: /var/log/ceph/cluster2-client.mirror.44739.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62617.log.gz 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44111.log 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44739.log.gz 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.42951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52151.log 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62175.log.gz 2026-02-18T06:45:05.637 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42951.log.gz 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47210.log 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52151.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.44111.log.gz 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51303.log 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47210.log: 87.2%gzip -- replaced with /var/log/ceph/cluster1-client.admin.52151.log.gz -5 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.32389.log 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.51303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34968.log 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.47210.log.gz 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51303.log.gz 2026-02-18T06:45:05.638 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63213.log 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34968.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32389.log.gz 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76373.log 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48303.log 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.34968.log.gz 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62724.log 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76373.log.gz 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63213.log.gz 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.48303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59297.log 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67578.log 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48303.log.gz 2026-02-18T06:45:05.639 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63866.log 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62724.log.gz 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48639.log 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67578.log.gz 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35052.log 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48639.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63866.log.gz 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34269.log 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.48639.log.gz 2026-02-18T06:45:05.640 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.78579.log 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.34269.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35052.log.gzgzip 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.48034.log 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.78579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41692.log 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48034.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79871.log 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.48034.log.gz 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61125.log 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34269.log.gz 2026-02-18T06:45:05.641 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.79871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47502.log 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.41692.log.gz 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79871.log.gz 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.68197.log 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.47502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68941.log 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61125.log.gz 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47502.log.gz 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.68197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53637.log 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.68941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.30896.log 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.53637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63010.log 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.78579.log.gz 2026-02-18T06:45:05.642 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.admin.68941.log.gz 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.30896.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53637.log.gz 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39656.log 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59297.log.gz 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29689.log 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.39656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53775.log 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63010.log.gz 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29689.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.39656.log.gz 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29689.log.gz 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78579.log 2026-02-18T06:45:05.643 INFO:teuthology.orchestra.run.trial130.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.68197.log.gz 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57522.log 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.78579.log: 88.6%gzip -- replaced with /var/log/ceph/cluster1-client.admin.53775.log.gz 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66820.log 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78579.log.gz 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64316.log 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.57522.log.gz 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66820.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.30896.log.gz 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62239.log 2026-02-18T06:45:05.644 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64316.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62903.log 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66820.log.gz 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62239.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64316.log.gz 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57491.log 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32043.log 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62903.log.gz 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72952.log 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32043.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.57491.log.gz 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73892.log 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32043.log.gz 2026-02-18T06:45:05.645 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72927.log 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44899.log 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72952.log.gz 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73892.log.gz 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63753.log 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72927.log.gz 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47094.log 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.44899.log.gz 2026-02-18T06:45:05.646 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64581.log 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.47094.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63753.log.gz 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44430.log 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55060.log 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr: 66.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64581.log.gz 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47094.log.gz 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.44430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70665.log 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.55060.log: -- replaced with /var/log/ceph/cluster2-client.admin.44430.log.gz 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28611.log 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55060.log.gz 2026-02-18T06:45:05.647 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79969.log 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70976.log 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28611.log.gz 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.79969.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62239.log.gz 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79969.log.gz 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59629.log 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.70976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64725.log 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70976.log.gz 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70665.log.gz 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71440.log 2026-02-18T06:45:05.648 INFO:teuthology.orchestra.run.trial130.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59629.log.gz 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46502.log 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64725.log.gz 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61344.log 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.46502.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71440.log.gz 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63320.log 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46502.log.gz 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31578.log 2026-02-18T06:45:05.649 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61344.log.gz 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50929.log 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63320.log: /var/log/ceph/cluster2-client.mirror.31578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33767.log 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.31578.log.gz 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.50929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60601.log 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63320.log.gz 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.33767.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.50929.log.gz 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.68030.log 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60601.log: gzip -5 --verbose 88.3% -- -- replaced with /var/log/ceph/cluster1-client.admin.33767.log.gz /var/log/ceph/cluster2-client.admin.52945.log 2026-02-18T06:45:05.650 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.68030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30454.log 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60601.log.gz 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.68030.log.gz 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52216.log 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30454.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.52945.log.gz 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43115.log 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr: 63.8% -- replaced with /var/log/ceph/cluster1-client.mirror.30454.log.gz 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59067.log 2026-02-18T06:45:05.651 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45165.log 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.52216.log.gz 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.59067.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.43115.log.gz 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62638.log 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.45165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79389.log 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.59067.log.gz 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45165.log.gz 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72664.log 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49212.log 2026-02-18T06:45:05.652 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.79389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79389.log.gz 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58555.log 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.49212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76148.log 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.58555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67777.log 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49212.log.gz 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58555.log.gz 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76148.log: 86.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.72664.log.gz -5 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.76878.log 2026-02-18T06:45:05.653 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.67777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61817.log 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76148.log.gz 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.76878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50192.log 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.67777.log.gz 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76878.log.gz 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46940.log 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61817.log.gz 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.50192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66687.log 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50192.log.gz 2026-02-18T06:45:05.654 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.46940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30761.log 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46940.log.gz 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33819.log 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53155.log 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr: 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30761.log.gz 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66687.log.gz 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.33819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63422.log 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62924.log 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.33819.log.gz 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.53155.log.gz/var/log/ceph/cluster2-client.mirror.63422.log: 2026-02-18T06:45:05.655 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44927.log 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45970.log 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63422.log.gz 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.44927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65251.log 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62638.log.gz 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.45970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36531.log 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44927.log.gz 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.65251.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45970.log.gz 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42355.log 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.65251.log.gz 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45603.log 2026-02-18T06:45:05.656 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.42355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72511.log 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.45603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29640.log 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.42355.log.gz 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45603.log.gz 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64598.log 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36531.log.gz 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.29640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31754.log 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29640.log.gz 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.64598.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64598.log.gz 2026-02-18T06:45:05.657 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77731.log 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56258.log 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72511.log.gz 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr: 88.2%/var/log/ceph/cluster2-client.admin.77731.log: -- replaced with /var/log/ceph/cluster2-client.mirror.31754.log.gz 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64256.log 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77731.log.gz 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.56258.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48490.log 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76758.log 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56258.log.gz 2026-02-18T06:45:05.658 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64256.log: /var/log/ceph/cluster2-client.admin.48490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63031.log 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64256.log.gz 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.48490.log.gz 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.76758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72828.log 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62924.log.gz 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47905.log 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76758.log.gz 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.72828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50522.log 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.72828.log.gz 2026-02-18T06:45:05.659 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.47905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.30791.log 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47905.log.gz 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55639.log 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.50522.log.gz 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.30791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46217.log 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.55639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30352.log 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.46217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38745.log 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55639.log.gz 2026-02-18T06:45:05.660 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52120.log 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46217.log.gz 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.30352.log: /var/log/ceph/cluster2-client.admin.38745.log: 72.2% -- replaced with /var/log/ceph/cluster1-client.admin.30352.log.gz 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74112.log 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52120.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.38745.log.gz 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63146.log 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74112.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.52120.log.gz 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58719.log 2026-02-18T06:45:05.661 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54440.log 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63146.log: /var/log/ceph/cluster1-client.admin.58719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31520.log 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63146.log.gz 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74112.log.gz 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.54440.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.58719.log.gz 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70630.log 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.54440.log.gz 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53430.log 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31520.log.gz 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63031.log.gz 2026-02-18T06:45:05.662 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.70630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61478.log 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.70630.log.gz 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30403.log 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61478.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.53430.log.gz 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67980.log 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63642.log 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30403.log.gz 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79358.log 2026-02-18T06:45:05.663 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35913.log 2026-02-18T06:45:05.664 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63642.log.gz 2026-02-18T06:45:05.664 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.79358.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74301.log 2026-02-18T06:45:05.664 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37893.log 2026-02-18T06:45:05.664 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74301.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.35913.log.gz 2026-02-18T06:45:05.664 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36362.log 2026-02-18T06:45:05.664 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74301.log.gz 2026-02-18T06:45:05.664 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30488.log 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.37893.log: /var/log/ceph/cluster2-client.mirror.36362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74091.log 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.37893.log.gz 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43776.log 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.30488.log.gz 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36362.log.gz 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64167.log 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74091.log.gz 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.43776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30369.log 2026-02-18T06:45:05.665 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64167.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43776.log.gz 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68241.log 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68670.log 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30369.log: 61.6% -- replaced with /var/log/ceph/cluster1-client.mirror.30369.log.gz 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.68241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49184.log 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61478.log.gz 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68241.log.gz 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64167.log.gz 2026-02-18T06:45:05.666 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.68670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34906.log 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67980.log.gz 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49184.log: 90.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.68670.log.gz 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62374.log 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79358.log.gz 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34906.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.49184.log.gz 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35365.log 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.34906.log.gz 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65282.log 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62374.log.gz 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61270.log 2026-02-18T06:45:05.667 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.65282.log: gzip 88.2% -5 -- replaced with /var/log/ceph/cluster1-client.admin.35365.log.gz --verbose 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr: -- /var/log/ceph/cluster1-client.admin.41373.log 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61270.log: 87.1%gzip -- replaced with /var/log/ceph/cluster1-client.admin.65282.log.gz -5 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.68992.log 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41373.log: 88.2%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.61270.log.gz 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30420.log 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.68992.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.41373.log.gz 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42630.log 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53609.log 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30420.log.gz 2026-02-18T06:45:05.668 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.42630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66974.log 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53609.log: gzip -5 --verbose 88.1% -- /var/log/ceph/cluster2-client.admin.50062.log -- replaced with /var/log/ceph/cluster2-client.admin.42630.log.gz 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.53609.log.gz 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33951.log 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68992.log.gz 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.50062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49608.log 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66974.log.gz 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33951.log: 88.1%gzip -- replaced with /var/log/ceph/cluster2-client.admin.50062.log.gz -5 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.65356.log 2026-02-18T06:45:05.669 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37793.log 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.65356.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.49608.log.gz 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74335.log 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65356.log.gz 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.37793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45444.log 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33951.log.gz 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37793.log.gz 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60068.log 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74335.log.gz 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65388.log 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.45444.log.gz 2026-02-18T06:45:05.670 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.60068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51268.log 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.65388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40630.log 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.60068.log.gz 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.51268.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.65388.log.gz 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32853.log 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51268.log.gz 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68727.log 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.32853.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.40630.log.gz 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41313.log 2026-02-18T06:45:05.671 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.68727.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.32853.log.gz 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76477.log 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.41313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70949.log 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52610.log 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41313.log.gz 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70949.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76477.log.gz 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68727.log.gz 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68842.log 2026-02-18T06:45:05.672 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77582.log 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.68842.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.52610.log.gz 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79229.log 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77582.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.68842.log.gz 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70949.log.gz 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73865.log 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77582.log.gz 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.79229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42986.log 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57621.log 2026-02-18T06:45:05.673 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.42986.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73865.log.gz 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67128.log 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42986.log.gz 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.57621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53301.log 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57621.log.gz 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67128.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33221.log 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.53301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72285.log 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53301.log.gz 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71585.log 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67128.log.gz 2026-02-18T06:45:05.674 INFO:teuthology.orchestra.run.trial130.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33221.log.gz 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35798.log 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72285.log.gz 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58771.log 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr: 68.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71585.log.gz 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67444.log 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.58771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51538.log 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.35798.log.gz 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58413.log 2026-02-18T06:45:05.675 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.51538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44461.log 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.51538.log.gz 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.58413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56417.log 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.58413.log.gz 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63378.log 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.44461.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67444.log.gz 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43294.log 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.44461.log.gz 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79229.log.gz 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.56417.log.gz 2026-02-18T06:45:05.676 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76849.log 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77879.log 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.43294.log.gz 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34289.log 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.30791.log.gz 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr: 87.1%/var/log/ceph/cluster2-client.mirror.77879.log: -- replaced with /var/log/ceph/cluster1-client.admin.76849.log.gz 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63664.log 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77879.log.gz 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63378.log.gz 2026-02-18T06:45:05.677 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.34289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77513.log 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.68074.log 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63664.log.gz 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71394.log 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77513.log.gz 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.68074.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34289.log.gz 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72107.log 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61290.log 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59930.log 2026-02-18T06:45:05.678 INFO:teuthology.orchestra.run.trial130.stderr: 65.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72107.log.gz 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58771.log.gz 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71359.log 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.59930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53708.log 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.68074.log.gz 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71394.log.gz 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58101.log 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.71359.log: /var/log/ceph/cluster1-client.mirror.53708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68810.log 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.71359.log.gz 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53708.log.gz 90.6% 2026-02-18T06:45:05.679 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61290.log.gz 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.58101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67062.log 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59930.log.gz 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58101.log.gz 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.68810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80361.log 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67062.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.68810.log.gz 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62682.log 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38855.log 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80361.log.gz 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67062.log.gz 2026-02-18T06:45:05.680 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62262.log 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72863.log 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62682.log.gz 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56792.log 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38855.log.gz 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62262.log.gz 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51044.log 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.69308.log 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.56792.log: /var/log/ceph/cluster1-client.admin.51044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70599.log 2026-02-18T06:45:05.681 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56792.log.gz 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.69308.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.51044.log.gz 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40433.log 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr: 19.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.69308.log.gz 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72863.log.gz 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.70599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39494.log 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- 88.1% /var/log/ceph/cluster1-client.mirror.0.69298.log 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.70599.log.gz 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40433.log: /var/log/ceph/cluster2-client.mirror.39494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75223.log 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.40433.log.gz 2026-02-18T06:45:05.682 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.69298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55759.log 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr: 17.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.69298.log.gz 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64687.log 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75223.log.gz 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60455.log 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39494.log.gz 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- 88.2% /var/log/ceph/cluster2-client.mirror.75000.log -- replaced with /var/log/ceph/cluster2-client.admin.55759.log.gz 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64687.log: 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64687.log.gz 2026-02-18T06:45:05.683 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42176.log 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72413.log 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60455.log.gz 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75000.log.gz 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.69351.log 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72413.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42176.log.gz 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46746.log 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.1.69351.log: -- replaced with /var/log/ceph/cluster2-client.mirror.72413.log.gz 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67040.log 2026-02-18T06:45:05.684 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54895.log 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46746.log: /var/log/ceph/cluster2-client.mirror.67040.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.46746.log.gz 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28353.log 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48244.log 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67040.log.gz 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28353.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54895.log.gz 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28353.log.gz 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60981.log 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59148.log 2026-02-18T06:45:05.685 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.48244.log.gz 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32136.log 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60981.log.gz 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.59148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69484.log 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32136.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29067.log 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.59148.log.gz 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32136.log.gz 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45703.log 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29067.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69484.log.gz 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33178.log 2026-02-18T06:45:05.686 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29067.log.gz 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52916.log 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.69351.log.gz 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.45703.log.gz 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38618.log 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79139.log 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28513.log 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.52916.log.gz 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38618.log.gz 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.79139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36739.log 2026-02-18T06:45:05.687 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.79139.log.gz 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54622.log 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28513.log.gz 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31260.log 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36739.log.gz 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.54622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67340.log 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54622.log.gz 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76736.log 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31260.log.gz 2026-02-18T06:45:05.688 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32882.log 2026-02-18T06:45:05.689 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.76736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.73368.log 2026-02-18T06:45:05.689 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67340.log.gz 2026-02-18T06:45:05.689 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.32882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73331.log 2026-02-18T06:45:05.689 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.32882.log.gz 2026-02-18T06:45:05.689 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.73368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72320.log 2026-02-18T06:45:05.689 INFO:teuthology.orchestra.run.trial130.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.73368.log.gz 2026-02-18T06:45:05.689 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76736.log.gz 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51626.log 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72320.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73331.log.gz 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36337.log 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72320.log.gz 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.51626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78072.log 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74952.log 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51626.log.gz 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.78072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73055.log 2026-02-18T06:45:05.690 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36337.log.gz 2026-02-18T06:45:05.691 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71743.log 2026-02-18T06:45:05.691 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28763.log 2026-02-18T06:45:05.691 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73055.log.gz 2026-02-18T06:45:05.691 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38778.log 2026-02-18T06:45:05.691 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71743.log: /var/log/ceph/cluster2-client.admin.28763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39687.log 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28763.log.gz 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71743.log.gz 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.38778.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74952.log.gz 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57990.log 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.38778.log.gz 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40404.log 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.39687.log.gz 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59516.log 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60183.log 87.2% 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.57990.log.gz 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.40404.log.gz 2026-02-18T06:45:05.692 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79413.log 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59448.log 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59516.log.gz 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.79413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67841.log 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79413.log.gz 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60183.log.gz 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36390.log 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.33178.log.gz 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.67841.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59448.log.gz 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75554.log 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36858.log 2026-02-18T06:45:05.693 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.admin.67841.log.gz 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36390.log.gz 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55699.log 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57183.log 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71551.log 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.57183.log: 94.0%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.78072.log.gz 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53458.log 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.55699.log.gz 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75554.log.gz 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57183.log.gz 2026-02-18T06:45:05.694 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71551.log: gzip -5 64.6% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.71551.log.gz 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr: /var/log/ceph/cluster1-client.mirror.59097.log 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.53458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72554.log 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72090.log 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.59097.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53458.log.gz 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.72554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65184.log 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59097.log.gz 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29983.log 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72554.log.gz 2026-02-18T06:45:05.695 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72090.log: 64.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72090.log.gz 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.65184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.77136.log 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65184.log.gz 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62791.log 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29983.log.gz 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.0.77136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35735.log 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66886.log 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62791.log.gz 2026-02-18T06:45:05.696 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35735.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59398.log 2026-02-18T06:45:05.697 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66886.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.35735.log.gz 2026-02-18T06:45:05.697 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37641.log 2026-02-18T06:45:05.697 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64129.log 2026-02-18T06:45:05.697 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66886.log.gz 2026-02-18T06:45:05.697 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36858.log.gz 2026-02-18T06:45:05.697 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.37641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73718.log 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40287.log 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.37641.log.gz 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64129.log.gz 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40808.log 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73718.log.gz/var/log/ceph/cluster1-client.mirror.40287.log: 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29493.log 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59398.log.gz 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31711.log 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29493.log: 88.2% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40808.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.29493.log.gz 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.698 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59581.log 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46308.log 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40287.log.gz 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75877.log 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33996.log 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.46308.log.gz 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76636.log 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31711.log.gz 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75877.log.gz 2026-02-18T06:45:05.699 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45353.log 2026-02-18T06:45:05.700 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.76636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56020.log 2026-02-18T06:45:05.700 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.45353.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76636.log.gz 2026-02-18T06:45:05.700 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52462.log 2026-02-18T06:45:05.700 INFO:teuthology.orchestra.run.trial130.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.77136.log.gz 2026-02-18T06:45:05.700 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56384.log 2026-02-18T06:45:05.700 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56020.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45353.log.gz 2026-02-18T06:45:05.700 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62197.log 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.56020.log.gz 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.52462.log.gz 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64540.log 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.56384.log.gz 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56565.log 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33657.log 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.64540.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62197.log.gz 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64540.log.gz 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.56565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69756.log 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.56565.log.gz 2026-02-18T06:45:05.701 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52036.log 2026-02-18T06:45:05.702 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33657.log.gz 2026-02-18T06:45:05.702 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77094.log 2026-02-18T06:45:05.702 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73093.log 2026-02-18T06:45:05.702 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69756.log.gz 2026-02-18T06:45:05.702 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.52036.log.gz 2026-02-18T06:45:05.702 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55025.log 2026-02-18T06:45:05.702 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.73093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37228.log 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.73093.log.gz 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33996.log.gz 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.55025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36764.log 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.37228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46277.log 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55025.log.gz 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77094.log.gz 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37228.log.gz 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.73404.log 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36764.log.gz 2026-02-18T06:45:05.703 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33283.log 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.73404.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.46277.log.gz 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50093.log 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr: 19.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.73404.log.gz 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59581.log.gz 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52641.log 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33283.log.gz 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.50093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62858.log 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.50093.log.gz 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74974.log 2026-02-18T06:45:05.704 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.52641.log.gz 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74581.log 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62858.log.gz 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48730.log 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69814.log 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74581.log.gz 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.48730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28448.log 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.48730.log.gz 2026-02-18T06:45:05.705 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60907.log/var/log/ceph/cluster2-client.mirror.69814.log: 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr: 62.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69814.log.gz 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36029.log 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.28448.log.gz 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74974.log.gz 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75378.log 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32171.log 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36029.log: /var/log/ceph/cluster1-client.mirror.75378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61647.log 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36029.log.gz 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75378.log.gz 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74235.log 2026-02-18T06:45:05.706 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32171.log.gz 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50550.log 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61647.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60907.log.gz 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74235.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47746.log 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61647.log.gz 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74235.log.gz 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr:gzip/var/log/ceph/cluster2-client.mirror.50550.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41960.log 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67714.log 2026-02-18T06:45:05.707 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.admin.47746.log.gz 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.41960.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50550.log.gz 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59910.log 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51717.log 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41960.log.gz 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67714.log.gz 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71887.log 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.51717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37167.log 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.71887.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.51717.log.gz 2026-02-18T06:45:05.708 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55926.log 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.37167.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.71887.log.gz 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32950.log 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59910.log.gz 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37167.log.gz 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.55926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40373.log 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55926.log.gz 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60235.log 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.40373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49639.log 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60235.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.40373.log.gz 2026-02-18T06:45:05.709 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44580.log 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61852.log 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32950.log.gz 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60235.log.gz 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.49639.log.gz 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70837.log 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.44580.log.gz 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76675.log 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61852.log.gz 2026-02-18T06:45:05.710 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.70837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45911.log 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70837.log.gz 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64094.log 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.76675.log.gz 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43748.log 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.45911.log.gz 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38121.log 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43748.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64094.log.gz 2026-02-18T06:45:05.711 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63488.log 2026-02-18T06:45:05.712 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.43748.log.gz 2026-02-18T06:45:05.712 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72797.log 2026-02-18T06:45:05.712 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.38121.log.gz 2026-02-18T06:45:05.712 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.63488.log: --verbose -- /var/log/ceph/cluster1-client.mirror.0.30782.log 2026-02-18T06:45:05.712 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.72797.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63488.log.gz 2026-02-18T06:45:05.712 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55818.log 2026-02-18T06:45:05.712 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.72797.log.gz 2026-02-18T06:45:05.713 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.30782.log: gzip 19.1% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.0.30782.log.gz -- 2026-02-18T06:45:05.713 INFO:teuthology.orchestra.run.trial130.stderr: /var/log/ceph/cluster1-client.mirror.45094.log 2026-02-18T06:45:05.713 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.55818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.77241.log 2026-02-18T06:45:05.713 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.45094.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55818.log.gz 2026-02-18T06:45:05.713 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.78617.log 2026-02-18T06:45:05.713 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.2.77241.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45094.log.gz 2026-02-18T06:45:05.713 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59844.log 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.78617.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.78617.log.gz 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29003.log 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.59844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73961.log 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.59844.log.gz 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr:gzip/var/log/ceph/cluster2-client.admin.29003.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32248.log 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29003.log.gz 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.73961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67491.log 2026-02-18T06:45:05.714 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.73961.log.gz 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52064.log 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32248.log.gz 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76068.log 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.77241.log.gz 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.52064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58234.log 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52883.log 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52064.log.gz 2026-02-18T06:45:05.715 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76068.log.gz 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.58234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35145.log 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.58234.log.gz 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.52883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77914.log 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.52883.log.gz 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47303.log 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77914.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.35145.log.gz 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44609.log 2026-02-18T06:45:05.716 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77914.log.gz 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.47303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40598.log 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.44609.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47303.log.gz 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30727.log 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.44609.log.gz 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.40598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33850.log 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40598.log.gz 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.30727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73930.log 67.9% 2026-02-18T06:45:05.717 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.30727.log.gz 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.33850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33715.log 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.73930.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.33850.log.gz 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45670.log 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67491.log.gz 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.73930.log.gz 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.33715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37581.log 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33715.log.gz 2026-02-18T06:45:05.718 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36891.log 2026-02-18T06:45:05.719 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.45670.log.gz 2026-02-18T06:45:05.719 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.37581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34626.log 2026-02-18T06:45:05.719 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75111.log 2026-02-18T06:45:05.719 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.37581.log.gz 2026-02-18T06:45:05.719 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36891.log.gz 2026-02-18T06:45:05.719 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57283.log 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.75111.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.34626.log.gz 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47467.log 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.75111.log.gz 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56656.log 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.57283.log.gz 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.47467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43538.log 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47467.log.gz 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31902.log 2026-02-18T06:45:05.720 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.56656.log.gz 2026-02-18T06:45:05.721 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70809.log 2026-02-18T06:45:05.721 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.43538.log.gz 2026-02-18T06:45:05.721 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.31902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67470.log 2026-02-18T06:45:05.721 INFO:teuthology.orchestra.run.trial130.stderr: 71.5% -- replaced with /var/log/ceph/cluster1-client.mirror.31902.log.gz 2026-02-18T06:45:05.721 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37700.log 2026-02-18T06:45:05.721 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31359.log 2026-02-18T06:45:05.721 INFO:teuthology.orchestra.run.trial130.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70809.log.gz 2026-02-18T06:45:05.722 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67470.log.gz 2026-02-18T06:45:05.722 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.37700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42568.log 2026-02-18T06:45:05.722 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31359.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37700.log.gz 2026-02-18T06:45:05.722 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60028.log 2026-02-18T06:45:05.722 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63843.log 2026-02-18T06:45:05.722 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.42568.log.gz 2026-02-18T06:45:05.722 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75923.log 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63843.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60028.log.gz 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61775.log 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63843.log.gz 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41807.log 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75923.log.gz 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40692.log 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61775.log.gz 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.41807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31613.log 2026-02-18T06:45:05.723 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.41807.log.gz 2026-02-18T06:45:05.724 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31226.log 2026-02-18T06:45:05.724 INFO:teuthology.orchestra.run.trial130.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.40692.log.gz 2026-02-18T06:45:05.724 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30437.log 2026-02-18T06:45:05.724 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31613.log.gz 2026-02-18T06:45:05.724 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31226.log: 71.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31226.log.gz 2026-02-18T06:45:05.724 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74016.log 2026-02-18T06:45:05.724 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30437.log: 67.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30437.log.gz 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59429.log 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63464.log 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.59429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62745.log 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31359.log.gz 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59429.log.gz 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.63464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73026.log 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74016.log.gz 2026-02-18T06:45:05.725 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40232.log 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.73026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64279.log 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73026.log.gz 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40232.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63464.log.gz 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39395.log 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.40232.log.gz 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.64279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57781.log 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64279.log.gz 2026-02-18T06:45:05.726 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.39395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51075.log 2026-02-18T06:45:05.727 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39395.log.gz 2026-02-18T06:45:05.727 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.57781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37828.log 2026-02-18T06:45:05.727 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.57781.log.gz 2026-02-18T06:45:05.727 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.51075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72883.log 2026-02-18T06:45:05.727 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.51075.log.gz 2026-02-18T06:45:05.727 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.37828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76551.log 2026-02-18T06:45:05.727 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37828.log.gz 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32822.log 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.76551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30505.log 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.admin.76551.log.gz 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.32822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72476.log 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.32822.log.gz 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30505.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72883.log.gz 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.30505.log.gz 2026-02-18T06:45:05.728 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54469.log 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.72476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48519.log 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62745.log.gz 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54469.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.72476.log.gz 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76106.log 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.54469.log.gz 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.48519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79106.log 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.48519.log.gz 2026-02-18T06:45:05.729 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31243.log 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.79106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60887.log 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr: 88.9%/var/log/ceph/cluster1-client.mirror.31243.log: -- replaced with /var/log/ceph/cluster1-client.admin.79106.log.gz 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38813.log 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr: 71.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31243.log.gz 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76106.log.gz 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42868.log 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.38813.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60887.log.gz 2026-02-18T06:45:05.730 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41723.log 2026-02-18T06:45:05.731 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.42868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33881.log 2026-02-18T06:45:05.731 INFO:teuthology.orchestra.run.trial130.stderr: 89.2%/var/log/ceph/cluster2-client.admin.41723.log: -- replaced with /var/log/ceph/cluster2-client.mirror.42868.log.gz 2026-02-18T06:45:05.731 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54500.log 2026-02-18T06:45:05.731 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.41723.log.gz 91.9% 2026-02-18T06:45:05.731 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38813.log.gz 2026-02-18T06:45:05.731 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.33881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56079.log 2026-02-18T06:45:05.731 INFO:teuthology.orchestra.run.trial130.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.admin.33881.log.gz 2026-02-18T06:45:05.732 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70760.log 2026-02-18T06:45:05.732 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.54500.log.gz 2026-02-18T06:45:05.732 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.56079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.78570.log 2026-02-18T06:45:05.732 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64636.log 2026-02-18T06:45:05.732 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70760.log.gz 2026-02-18T06:45:05.732 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56079.log.gz 2026-02-18T06:45:05.732 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.78570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36282.log 2026-02-18T06:45:05.733 INFO:teuthology.orchestra.run.trial130.stderr: 19.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.78570.log.gz 2026-02-18T06:45:05.733 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.64636.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.64636.log.gz 2026-02-18T06:45:05.733 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49883.log 2026-02-18T06:45:05.733 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29199.log 2026-02-18T06:45:05.733 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49883.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36282.log.gz 2026-02-18T06:45:05.733 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44142.log 2026-02-18T06:45:05.733 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.49883.log.gz 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33124.log 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29199.log.gz 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48421.log 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.44142.log.gz 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53337.log 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33124.log.gz 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.48421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32568.log 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48421.log.gz 2026-02-18T06:45:05.734 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79920.log 2026-02-18T06:45:05.735 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.53337.log.gz 2026-02-18T06:45:05.735 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32568.log: gzip -5 --verbose -- 67.9% /var/log/ceph/cluster2-client.admin.29248.log 2026-02-18T06:45:05.735 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.32568.log.gz 2026-02-18T06:45:05.735 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.79920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52847.log 2026-02-18T06:45:05.735 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79920.log.gz 2026-02-18T06:45:05.735 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29248.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29248.log.gz 2026-02-18T06:45:05.735 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39536.log 2026-02-18T06:45:05.736 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.52847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46534.log 2026-02-18T06:45:05.736 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52847.log.gz 2026-02-18T06:45:05.736 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67295.log 2026-02-18T06:45:05.736 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39536.log.gz 2026-02-18T06:45:05.736 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.46534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61210.log 2026-02-18T06:45:05.736 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.67295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62484.log 2026-02-18T06:45:05.736 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.46534.log.gz 2026-02-18T06:45:05.737 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61210.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67295.log.gz 2026-02-18T06:45:05.737 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56446.log 2026-02-18T06:45:05.737 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41032.log 2026-02-18T06:45:05.737 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62484.log.gz 2026-02-18T06:45:05.737 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75900.log 2026-02-18T06:45:05.737 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.56446.log.gz 2026-02-18T06:45:05.737 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.41032.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64495.log 2026-02-18T06:45:05.738 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41032.log.gz 2026-02-18T06:45:05.738 INFO:teuthology.orchestra.run.trial130.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61210.log.gz 2026-02-18T06:45:05.738 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.75900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71460.log 2026-02-18T06:45:05.738 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75900.log.gz 2026-02-18T06:45:05.738 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49296.log 2026-02-18T06:45:05.738 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77946.log 2026-02-18T06:45:05.738 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.49296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45504.log 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64495.log.gz 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.77946.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49296.log.gz 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49093.log 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.77946.log.gz 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71460.log.gz 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28423.log 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.49093.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.45504.log.gz 2026-02-18T06:45:05.739 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.37230.log 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.49093.log.gz 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28423.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28423.log.gz 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61892.log 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.3.37230.log: 19.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.37230.log.gz 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42532.log 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.61892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42601.log 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.61892.log.gz 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.42532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67204.log 2026-02-18T06:45:05.740 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42532.log.gz 2026-02-18T06:45:05.741 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47806.log 2026-02-18T06:45:05.741 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.42601.log.gz 2026-02-18T06:45:05.741 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.67204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75426.log 2026-02-18T06:45:05.741 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.67204.log.gz 2026-02-18T06:45:05.741 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43657.log 2026-02-18T06:45:05.741 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75426.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.47806.log.gz 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30548.log 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75426.log.gz 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50320.log 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.43657.log.gz 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.30548.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30548.log.gz 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62439.log 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.50320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43143.log 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50320.log.gz 2026-02-18T06:45:05.742 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.62439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39048.log 2026-02-18T06:45:05.743 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.43143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42235.log 2026-02-18T06:45:05.743 INFO:teuthology.orchestra.run.trial130.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62439.log.gz 2026-02-18T06:45:05.743 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61554.log 2026-02-18T06:45:05.743 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43143.log.gz 2026-02-18T06:45:05.743 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39048.log.gz/var/log/ceph/cluster2-client.mirror.42235.log: 2026-02-18T06:45:05.743 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54926.log 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.61554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.73367.log 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42235.log.gz 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61554.log.gz 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.54926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42780.log 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.54926.log.gz 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.0.73367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38960.log 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41067.log 2026-02-18T06:45:05.744 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.42780.log.gz 2026-02-18T06:45:05.745 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38181.log 2026-02-18T06:45:05.745 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38960.log.gz 2026-02-18T06:45:05.745 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.41067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33971.log 2026-02-18T06:45:05.745 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41067.log.gz 2026-02-18T06:45:05.745 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.38181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43597.log 2026-02-18T06:45:05.745 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.38181.log.gz 2026-02-18T06:45:05.745 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.33971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60809.log 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.43597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42749.log 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.60809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68039.log 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43597.log.gz 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60809.log.gz 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33971.log.gz 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.42749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52393.log 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.42749.log.gz 2026-02-18T06:45:05.746 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.68039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74604.log 2026-02-18T06:45:05.747 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.52393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28908.log 2026-02-18T06:45:05.747 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52393.log.gz 2026-02-18T06:45:05.747 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39295.log 2026-02-18T06:45:05.747 INFO:teuthology.orchestra.run.trial130.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74604.log.gz 2026-02-18T06:45:05.747 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28908.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28908.log.gz 2026-02-18T06:45:05.747 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49153.log 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75031.log 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39295.log.gz 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77309.log 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.49153.log.gz 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.68040.log 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68039.log.gz 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.77309.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75031.log.gz 2026-02-18T06:45:05.748 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62029.log 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr: 86.2% -- replaced with /var/log/ceph/cluster1-client.mirror.77309.log.gz 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.68040.log: 19.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.68040.log.gz 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63619.log 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46396.log 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.63619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30386.log 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63619.log.gz 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.46396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68972.log 2026-02-18T06:45:05.749 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62029.log.gz 2026-02-18T06:45:05.750 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.30386.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46396.log.gz 2026-02-18T06:45:05.750 INFO:teuthology.orchestra.run.trial130.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30386.log.gz 2026-02-18T06:45:05.750 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.77309.log 2026-02-18T06:45:05.750 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.68972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66754.log 2026-02-18T06:45:05.750 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68972.log.gz 2026-02-18T06:45:05.750 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.3.77309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74725.log 2026-02-18T06:45:05.750 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.37124.log 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32551.log 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66754.log.gz 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74725.log.gz 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.1.37124.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.37124.log.gz 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42661.log 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55580.log 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr: 67.7% -- replaced with /var/log/ceph/cluster1-client.mirror.32551.log.gz 2026-02-18T06:45:05.751 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.42661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80312.log 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.42661.log.gz 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55580.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.77309.log.gz 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55991.log 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.55580.log.gz 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78011.log 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80312.log.gz 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69955.log 2026-02-18T06:45:05.752 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.55991.log.gz 2026-02-18T06:45:05.753 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.78011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57371.log 2026-02-18T06:45:05.753 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.78011.log.gz 2026-02-18T06:45:05.753 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38833.log 2026-02-18T06:45:05.753 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69955.log.gz 2026-02-18T06:45:05.753 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.57371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25567.log 2026-02-18T06:45:05.753 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.38833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38929.log 2026-02-18T06:45:05.753 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57371.log.gz 2026-02-18T06:45:05.754 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.25567.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25567.log.gz 2026-02-18T06:45:05.754 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62879.log 2026-02-18T06:45:05.754 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69480.log 2026-02-18T06:45:05.754 INFO:teuthology.orchestra.run.trial130.stderr: 91.9% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38833.log.gz 2026-02-18T06:45:05.754 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.38929.log.gz 2026-02-18T06:45:05.754 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75646.log 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.69480.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62879.log.gz 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr:gzip 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69480.log.gz -5 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.77632.log 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36415.log 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75646.log.gz 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77632.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77632.log.gz 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52358.log 2026-02-18T06:45:05.755 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.36415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66710.log 2026-02-18T06:45:05.756 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.36415.log.gz 2026-02-18T06:45:05.756 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.52358.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39787.log 2026-02-18T06:45:05.756 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52358.log.gz 2026-02-18T06:45:05.756 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.66710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59181.log 2026-02-18T06:45:05.756 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.39787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48758.log 2026-02-18T06:45:05.756 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66710.log.gz 2026-02-18T06:45:05.756 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.59181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52670.log 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.48758.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.59181.log.gz 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33689.log 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.52670.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48758.log.gz 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35396.log 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.52670.log.gz 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.33689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47596.log 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.33689.log.gz 2026-02-18T06:45:05.757 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62768.log 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.47596.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.35396.log.gz 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50121.log 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.47596.log.gz 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.62768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46129.log 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62768.log.gz 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.50121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34741.log 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50121.log.gz 2026-02-18T06:45:05.758 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.46129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45294.log 2026-02-18T06:45:05.759 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.46129.log.gz 2026-02-18T06:45:05.759 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53064.log 2026-02-18T06:45:05.759 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.45294.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.34741.log.gz 2026-02-18T06:45:05.759 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57809.log 2026-02-18T06:45:05.759 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.45294.log.gz 2026-02-18T06:45:05.759 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.53064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28909.log 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.57809.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.53064.log.gz 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73524.log 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39787.log.gz 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.28909.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28909.log.gz 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43894.log 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57809.log.gz 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.73524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71619.log 2026-02-18T06:45:05.760 INFO:teuthology.orchestra.run.trial130.stderr: 81.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73524.log.gz 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.43894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72046.log 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43894.log.gz 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.71619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50285.log 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71619.log.gz 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67319.log 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72046.log.gz 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.50285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.33179.log 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50285.log.gz 2026-02-18T06:45:05.761 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41225.log 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.33179.log: 17.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.33179.log.gz 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53578.log 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67319.log.gz 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.41225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33360.log 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.41225.log.gz 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.53578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71490.log 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.53578.log.gz 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.33360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59320.log 2026-02-18T06:45:05.762 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.71490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39108.log 2026-02-18T06:45:05.763 INFO:teuthology.orchestra.run.trial130.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71490.log.gz 2026-02-18T06:45:05.763 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.59320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69684.log 2026-02-18T06:45:05.763 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59320.log.gz 2026-02-18T06:45:05.763 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58995.log 2026-02-18T06:45:05.763 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47534.log 2026-02-18T06:45:05.763 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69684.log.gz 2026-02-18T06:45:05.764 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.58995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67746.log 2026-02-18T06:45:05.764 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.47534.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.58995.log.gz 2026-02-18T06:45:05.764 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53183.log 2026-02-18T06:45:05.764 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.47534.log.gz 2026-02-18T06:45:05.764 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.67746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68601.log 2026-02-18T06:45:05.764 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.53183.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.67746.log.gz 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49785.log 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33360.log.gz 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53183.log.gz 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.68601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54320.log 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.49785.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.68601.log.gz 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50729.log 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49785.log.gz 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.54320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64704.log 2026-02-18T06:45:05.765 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.54320.log.gz 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.50729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.78614.log 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr: 89.2%/var/log/ceph/cluster2-client.mirror.64704.log: -- replaced with /var/log/ceph/cluster1-client.mirror.50729.log.gz 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.69353.log 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64704.log.gz 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.1.78614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43509.log 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39108.log.gz 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.69353.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.69353.log.gz 2026-02-18T06:45:05.766 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49914.log 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55364.log 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.43509.log.gz 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.49914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35579.log 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.49914.log.gz 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29542.log 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.55364.log.gz 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.35579.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.78614.log.gz 2026-02-18T06:45:05.767 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40563.log 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.29542.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.35579.log.gz 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29542.log.gz 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44301.log 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.40563.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80018.log 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40563.log.gz 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.44301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77759.log 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44301.log.gz 2026-02-18T06:45:05.768 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80018.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80018.log.gz 2026-02-18T06:45:05.769 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48876.log 2026-02-18T06:45:05.769 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.77759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36714.log 2026-02-18T06:45:05.769 INFO:teuthology.orchestra.run.trial130.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77759.log.gz 2026-02-18T06:45:05.769 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.48876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40868.log 2026-02-18T06:45:05.769 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48876.log.gz 2026-02-18T06:45:05.769 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75534.log 2026-02-18T06:45:05.769 INFO:teuthology.orchestra.run.trial130.stderr: 25.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36714.log.gz 2026-02-18T06:45:05.770 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.33283.log 2026-02-18T06:45:05.770 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.40868.log.gz 2026-02-18T06:45:05.770 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.75534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55790.log 2026-02-18T06:45:05.770 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.33283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77977.log 2026-02-18T06:45:05.770 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.55790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55392.log 2026-02-18T06:45:05.770 INFO:teuthology.orchestra.run.trial130.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75534.log.gz 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.55790.log.gz 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.77977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64653.log 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.77977.log.gz 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.55392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51746.log 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64653.log: 66.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64653.log.gz 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr:gzip 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55392.log.gz -5 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.58964.log 2026-02-18T06:45:05.771 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.51746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61980.log 2026-02-18T06:45:05.772 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.51746.log.gz 2026-02-18T06:45:05.772 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.58964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73996.log 2026-02-18T06:45:05.772 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.58964.log.gz 2026-02-18T06:45:05.772 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.61980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56505.log 2026-02-18T06:45:05.772 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.73996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.37114.log 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.56505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72996.log 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61980.log.gz 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.0.37114.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.37114.log.gz 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80116.log 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56505.log.gz 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51777.log 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73996.log.gz 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72996.log.gz 2026-02-18T06:45:05.773 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80116.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80116.log.gz 2026-02-18T06:45:05.774 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56824.log 2026-02-18T06:45:05.774 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.51777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34449.log 2026-02-18T06:45:05.774 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.51777.log.gz 2026-02-18T06:45:05.774 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.56824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43405.log 2026-02-18T06:45:05.774 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.56824.log.gz 2026-02-18T06:45:05.774 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38366.log 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.34449.log.gz 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.43405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38335.log 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33178.log.gz 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43405.log.gz 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70395.log 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.38366.log.gz 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.38335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74931.log 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.38335.log.gz 2026-02-18T06:45:05.775 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.70395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80263.log 2026-02-18T06:45:05.776 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70395.log.gz 2026-02-18T06:45:05.776 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37082.log 2026-02-18T06:45:05.776 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80263.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80263.log.gz 2026-02-18T06:45:05.776 INFO:teuthology.orchestra.run.trial130.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74931.log.gz 2026-02-18T06:45:05.776 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34533.log 2026-02-18T06:45:05.776 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.37082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72632.log 2026-02-18T06:45:05.776 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.37082.log.gz 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43476.log 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34533.log.gz 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.72632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.68139.log 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72632.log.gz 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39218.log 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.43476.log.gz 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.68139.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.68139.log.gz 2026-02-18T06:45:05.777 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55213.log 2026-02-18T06:45:05.778 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.39218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37051.log 2026-02-18T06:45:05.778 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39218.log.gz 2026-02-18T06:45:05.778 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.55213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51976.log 2026-02-18T06:45:05.778 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.37051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64768.log 2026-02-18T06:45:05.778 INFO:teuthology.orchestra.run.trial130.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55213.log.gz 2026-02-18T06:45:05.778 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.37051.log.gz 2026-02-18T06:45:05.779 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.51976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36164.log 2026-02-18T06:45:05.779 INFO:teuthology.orchestra.run.trial130.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.51976.log.gz 2026-02-18T06:45:05.779 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.64768.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58444.log 2026-02-18T06:45:05.779 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.36164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68525.log 2026-02-18T06:45:05.779 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.36164.log.gz 2026-02-18T06:45:05.779 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.58444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34232.log 2026-02-18T06:45:05.779 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.58444.log.gz 2026-02-18T06:45:05.780 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.68525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32409.log 2026-02-18T06:45:05.780 INFO:teuthology.orchestra.run.trial130.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.68525.log.gz 2026-02-18T06:45:05.780 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36984.log 2026-02-18T06:45:05.780 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.32409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51598.log 2026-02-18T06:45:05.780 INFO:teuthology.orchestra.run.trial130.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.34232.log.gz 2026-02-18T06:45:05.780 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.36984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58932.log 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36984.log.gz 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.51598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57721.log 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.51598.log.gz 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.58932.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32409.log.gz 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39362.log 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58932.log.gz 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.57721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80408.log 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.57721.log.gz 2026-02-18T06:45:05.781 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.39362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71010.log 2026-02-18T06:45:05.782 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.39362.log.gz 2026-02-18T06:45:05.782 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.80408.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80408.log.gz 2026-02-18T06:45:05.782 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56684.log 2026-02-18T06:45:05.782 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.71010.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64768.log.gz 2026-02-18T06:45:05.782 INFO:teuthology.orchestra.run.trial130.stderr: 71.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71010.log.gz 2026-02-18T06:45:05.782 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70884.log 2026-02-18T06:45:05.782 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.56684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74900.log 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56684.log.gz 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.70884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30335.log 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70884.log.gz 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.74900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60831.log 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74900.log.gz 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.30335.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.30335.log.gz 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.30827.log 2026-02-18T06:45:05.783 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.60831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43203.log 2026-02-18T06:45:05.784 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60831.log.gz 2026-02-18T06:45:05.784 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.2.30827.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.30827.log.gz 2026-02-18T06:45:05.784 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53004.log 2026-02-18T06:45:05.784 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.43203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50641.log 2026-02-18T06:45:05.784 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.43203.log.gz 2026-02-18T06:45:05.784 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.53004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.77127.log 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.50641.log: gzip -5 --verbose -- 94.8% /var/log/ceph/cluster2-client.mirror.56757.log 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.33283.log.gz 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53004.log.gz 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.50641.log.gz 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.0.77127.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.77127.log.gz 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69272.log 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.56757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37612.log 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56757.log.gz 2026-02-18T06:45:05.785 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.69272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2026-02-18T06:45:05.786 INFO:teuthology.orchestra.run.trial130.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69272.log.gz 2026-02-18T06:45:05.786 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.37612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67018.log 2026-02-18T06:45:05.786 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.37612.log.gz 2026-02-18T06:45:05.786 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58472.log 2026-02-18T06:45:05.786 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.67018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40267.log 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.58472.log: gzip 89.3% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.67018.log.gz --verbose 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr: -- /var/log/ceph/cluster1-client.mirror.74467.log 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr: 89.1%/var/log/ceph/cluster2-client.mirror.40267.log: -- replaced with /var/log/ceph/cluster2-client.mirror.58472.log.gz 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44868.log 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.74467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50960.log 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74467.log.gz 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.44868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31392.log 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr: 91.4% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40267.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.44868.log.gz 2026-02-18T06:45:05.787 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:05.788 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.50960.log: -- /var/log/ceph/cluster1-client.admin.45882.log 2026-02-18T06:45:05.788 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.31392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.33360.log 2026-02-18T06:45:05.788 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.admin.50960.log.gz 2026-02-18T06:45:05.788 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31392.log.gz 2026-02-18T06:45:05.788 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45232.log 2026-02-18T06:45:05.788 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.45882.log.gz 2026-02-18T06:45:05.788 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.3.33360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43263.log 2026-02-18T06:45:05.789 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.45232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55486.log 2026-02-18T06:45:05.789 INFO:teuthology.orchestra.run.trial130.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.45232.log.gz 2026-02-18T06:45:05.789 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.43263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25694.log 2026-02-18T06:45:05.789 INFO:teuthology.orchestra.run.trial130.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.43263.log.gz 2026-02-18T06:45:05.789 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.55486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32366.log 2026-02-18T06:45:05.789 INFO:teuthology.orchestra.run.trial130.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55486.log.gz 2026-02-18T06:45:05.790 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.admin.25694.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25694.log.gz 2026-02-18T06:45:05.790 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40961.log 2026-02-18T06:45:05.790 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.mirror.32366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76497.log 2026-02-18T06:45:05.790 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.40961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55304.log 2026-02-18T06:45:05.790 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster2-client.mirror.76497.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40961.log.gz 2026-02-18T06:45:05.790 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34418.log 2026-02-18T06:45:05.791 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.55304.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32366.log.gz 2026-02-18T06:45:05.791 INFO:teuthology.orchestra.run.trial130.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.55304.log.gz 2026-02-18T06:45:05.791 INFO:teuthology.orchestra.run.trial130.stderr:/var/log/ceph/cluster1-client.admin.34418.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34418.log.gz 2026-02-18T06:45:05.791 INFO:teuthology.orchestra.run.trial130.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76497.log.gz 2026-02-18T06:45:05.806 INFO:teuthology.orchestra.run.trial130.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.33360.log.gz 2026-02-18T06:45:05.904 INFO:teuthology.orchestra.run.trial031.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2026-02-18T06:45:06.100 INFO:teuthology.orchestra.run.trial130.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2026-02-18T06:45:06.115 INFO:teuthology.orchestra.run.trial130.stderr: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.73367.log.gz 2026-02-18T06:45:08.670 INFO:teuthology.orchestra.run.trial031.stderr: 94.7% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2026-02-18T06:45:09.234 INFO:teuthology.orchestra.run.trial130.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2026-02-18T06:45:09.292 INFO:teuthology.orchestra.run.trial031.stderr: 94.8% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2026-02-18T06:45:09.347 INFO:teuthology.orchestra.run.trial130.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2026-02-18T06:45:09.711 INFO:teuthology.orchestra.run.trial031.stderr: 95.0% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2026-02-18T06:45:09.712 INFO:teuthology.orchestra.run.trial031.stderr: 2026-02-18T06:45:09.712 INFO:teuthology.orchestra.run.trial031.stderr:real 0m4.321s 2026-02-18T06:45:09.712 INFO:teuthology.orchestra.run.trial031.stderr:user 0m11.601s 2026-02-18T06:45:09.712 INFO:teuthology.orchestra.run.trial031.stderr:sys 0m0.575s 2026-02-18T06:45:10.141 INFO:teuthology.orchestra.run.trial130.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2026-02-18T06:45:10.142 INFO:teuthology.orchestra.run.trial130.stderr: 2026-02-18T06:45:10.142 INFO:teuthology.orchestra.run.trial130.stderr:real 0m4.729s 2026-02-18T06:45:10.142 INFO:teuthology.orchestra.run.trial130.stderr:user 0m13.377s 2026-02-18T06:45:10.142 INFO:teuthology.orchestra.run.trial130.stderr:sys 0m1.305s 2026-02-18T06:45:10.143 INFO:tasks.ceph:Archiving logs... 2026-02-18T06:45:10.144 DEBUG:teuthology.misc:Transferring archived files from trial031:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/remote/trial031/log 2026-02-18T06:45:10.145 DEBUG:teuthology.orchestra.run.trial031:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-18T06:45:11.074 DEBUG:teuthology.misc:Transferring archived files from trial130:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/remote/trial130/log 2026-02-18T06:45:11.075 DEBUG:teuthology.orchestra.run.trial130:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-18T06:45:12.824 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-18T06:45:12.860 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-18T06:45:12.860 DEBUG:teuthology.orchestra.run.trial031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-18T06:45:12.862 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-18T06:45:13.028 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-18T06:45:13.038 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-18T06:45:13.038 DEBUG:teuthology.orchestra.run.trial031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-18T06:45:13.040 DEBUG:teuthology.orchestra.run.trial130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-18T06:45:13.054 INFO:teuthology.orchestra.run.trial031.stderr:bash: line 1: ntpq: command not found 2026-02-18T06:45:13.056 INFO:teuthology.orchestra.run.trial130.stderr:bash: line 1: ntpq: command not found 2026-02-18T06:45:13.076 INFO:teuthology.orchestra.run.trial031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-18T06:45:13.076 INFO:teuthology.orchestra.run.trial031.stdout:=============================================================================== 2026-02-18T06:45:13.076 INFO:teuthology.orchestra.run.trial031.stdout:^+ li1150-42.members.linode> 2 6 377 28 -854us[ -854us] +/- 78ms 2026-02-18T06:45:13.076 INFO:teuthology.orchestra.run.trial031.stdout:^+ ntp.maxhost.io 2 7 377 96 -1562us[-1575us] +/- 56ms 2026-02-18T06:45:13.077 INFO:teuthology.orchestra.run.trial031.stdout:^+ 69.9.131.124 2 7 377 96 +5041us[+5028us] +/- 61ms 2026-02-18T06:45:13.077 INFO:teuthology.orchestra.run.trial031.stdout:^* time.richie.net 2 6 377 29 -1326us[-1335us] +/- 25ms 2026-02-18T06:45:13.078 INFO:teuthology.orchestra.run.trial130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-18T06:45:13.078 INFO:teuthology.orchestra.run.trial130.stdout:=============================================================================== 2026-02-18T06:45:13.078 INFO:teuthology.orchestra.run.trial130.stdout:^- 72-14-182-49.ip.linodeus> 2 7 377 96 +2375us[+2368us] +/- 60ms 2026-02-18T06:45:13.078 INFO:teuthology.orchestra.run.trial130.stdout:^- 72-14-186-59.ip.linodeus> 2 6 377 27 -2609us[-2609us] +/- 89ms 2026-02-18T06:45:13.078 INFO:teuthology.orchestra.run.trial130.stdout:^* ntp2.wiktel.com 1 6 377 29 -389us[ -395us] +/- 20ms 2026-02-18T06:45:13.078 INFO:teuthology.orchestra.run.trial130.stdout:^+ dns-e.ns4v.icu 2 7 377 33 +854us[ +848us] +/- 65ms 2026-02-18T06:45:13.079 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-18T06:45:13.088 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-18T06:45:13.089 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-18T06:45:13.101 DEBUG:teuthology.orchestra.run.trial031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-18T06:45:13.161 DEBUG:teuthology.orchestra.run.trial130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-18T06:45:13.205 DEBUG:teuthology.orchestra.run.trial031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-18T06:45:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:45:13.231 DEBUG:teuthology.orchestra.run.trial130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-18T06:45:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:45:13.256 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-18T06:45:13.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-18T06:45:13.282 INFO:teuthology.task.internal:Duration was 1663.559068 seconds 2026-02-18T06:45:13.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-18T06:45:13.299 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-18T06:45:13.300 DEBUG:teuthology.orchestra.run.trial031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-18T06:45:13.302 DEBUG:teuthology.orchestra.run.trial130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-18T06:45:13.337 INFO:teuthology.orchestra.run.trial031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-18T06:45:13.338 INFO:teuthology.orchestra.run.trial130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-18T06:45:13.589 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-18T06:45:13.589 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial031.front.sepia.ceph.com 2026-02-18T06:45:13.589 DEBUG:teuthology.orchestra.run.trial031:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-18T06:45:13.609 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial130.front.sepia.ceph.com 2026-02-18T06:45:13.609 DEBUG:teuthology.orchestra.run.trial130:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-18T06:45:13.647 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-18T06:45:13.647 DEBUG:teuthology.orchestra.run.trial031:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-18T06:45:13.651 DEBUG:teuthology.orchestra.run.trial130:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-18T06:45:13.728 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-18T06:45:13.728 DEBUG:teuthology.orchestra.run.trial031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-18T06:45:13.733 DEBUG:teuthology.orchestra.run.trial130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-18T06:45:13.756 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T06:45:13.756 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T06:45:13.756 INFO:teuthology.orchestra.run.trial031.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-18T06:45:13.756 INFO:teuthology.orchestra.run.trial031.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-18T06:45:13.757 INFO:teuthology.orchestra.run.trial031.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-18T06:45:13.760 INFO:teuthology.orchestra.run.trial031.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 88.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-18T06:45:13.792 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T06:45:13.792 INFO:teuthology.orchestra.run.trial130.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T06:45:13.792 INFO:teuthology.orchestra.run.trial130.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gzgzip 2026-02-18T06:45:13.792 INFO:teuthology.orchestra.run.trial130.stderr: -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-18T06:45:13.793 INFO:teuthology.orchestra.run.trial130.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-18T06:45:13.797 INFO:teuthology.orchestra.run.trial130.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 88.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-18T06:45:13.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-18T06:45:13.812 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-18T06:45:13.812 DEBUG:teuthology.orchestra.run.trial031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-18T06:45:13.837 DEBUG:teuthology.orchestra.run.trial130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-18T06:45:13.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-18T06:45:13.877 DEBUG:teuthology.orchestra.run.trial031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-18T06:45:13.880 DEBUG:teuthology.orchestra.run.trial130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-18T06:45:13.902 INFO:teuthology.orchestra.run.trial031.stdout:kernel.core_pattern = core 2026-02-18T06:45:13.926 INFO:teuthology.orchestra.run.trial130.stdout:kernel.core_pattern = core 2026-02-18T06:45:13.940 DEBUG:teuthology.orchestra.run.trial031:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-18T06:45:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:45:13.976 DEBUG:teuthology.orchestra.run.trial130:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-18T06:45:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T06:45:13.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-18T06:45:14.009 INFO:teuthology.task.internal:Transferring archived files... 2026-02-18T06:45:14.010 DEBUG:teuthology.misc:Transferring archived files from trial031:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/remote/trial031 2026-02-18T06:45:14.010 DEBUG:teuthology.orchestra.run.trial031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-18T06:45:14.047 DEBUG:teuthology.misc:Transferring archived files from trial130:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:26:40-rbd-20.2.1_2.17.26-baseline_2-distro-default-trial/54885/remote/trial130 2026-02-18T06:45:14.048 DEBUG:teuthology.orchestra.run.trial130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-18T06:45:14.079 INFO:teuthology.task.internal:Removing archive directory... 2026-02-18T06:45:14.080 DEBUG:teuthology.orchestra.run.trial031:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-18T06:45:14.082 DEBUG:teuthology.orchestra.run.trial130:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-18T06:45:14.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-18T06:45:14.148 INFO:teuthology.task.internal:Not uploading archives. 2026-02-18T06:45:14.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-18T06:45:14.165 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-18T06:45:14.165 DEBUG:teuthology.orchestra.run.trial031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-18T06:45:14.168 DEBUG:teuthology.orchestra.run.trial130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-18T06:45:14.184 INFO:teuthology.orchestra.run.trial031.stdout: 791912 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 18 06:45 /home/ubuntu/cephtest 2026-02-18T06:45:14.185 INFO:teuthology.orchestra.run.trial130.stdout: 786514 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 18 06:45 /home/ubuntu/cephtest 2026-02-18T06:45:14.186 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-18T06:45:14.205 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-18T06:45:14.245 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-policy-none} duration: 1663.559068441391 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-18T06:45:14.245 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-18T06:45:14.445 INFO:teuthology.run:pass